Bank Marketing Data Notebook

---

---

Table of Contents

• The Project
  • Business Problem
  • Methodology
• About the Dataset
  • Inputs
• Exploratory Data Analysis
  • Categorical Features
  • Continuous Features
• Data Pipeline & Preprocessing
  • Baseline
  • Feature Engineering
• Modeling
  • Tuning Models
  • Training Best Models
  • Feature Importance
  • Saving Model
• Conclusion

---

---

The Project

This notebook documents the Data Science process for the Tech Challenge 3 belonging to FIAP's Machine Learning Engineering PostGrad program.

In this project we will use the Bank Marketing dataset available at the UC Irvine Machine Learning Repository to build a model using supervised learning techniques for classification tasks.

Business Problem

A bank wants to improve the efficiency of its telemarketing campaigns for term deposits. Our goal is to create a machine learning model that can predict which clients are more likely to subscribe, allowing the bank to focus its efforts on the most promising leads, increasing the success rate and reducing marketing costs.

Methodology

This project follows a structured Machine Learning Workflow:

1. Exploratory Data Analysis: We will analyze the dataset to understand distributions, find patterns, and form our initial hypotheses.
   
   2.Data Pipeline & Preprocessing: We will build a simple classification model using Logistic Regression to build a baseline to compare our more complex models with. In this stage, we will also clean the data and engineer features based on insights from the exploratory analysis.
   
   
2. Modeling & Comparison: We will evaluate a suite of classifiers using cross-validation. Based on performance, we will select a few models for hyperparameter tuning with Optuna to extract the best performance possible.
   
   
3. Conclusion: In the conclusion, we will provide a final overview of our discoveries, models, and methodology, focusing on how this project can help with the solution of the business problem.

---

---

About the Dataset

The data is related to direct marketing campaigns of a Portuguese banking institution. These campaigns were primarily conducted via phone calls, and often, more than one contact to the same client was required to determine if the product (a bank term deposit) would be subscribed to.

The classification goal of this project is to predict if a client will subscribe (yes or no) to a term deposit, which is our target variable, y.

Inputs

To better understand the content of this dataset, we listed below the variable names, types, and description, as found in the variables table provided on the official UC Irvine Machine Learning Repository:

Variable Name	Type	Description
age	Integer	The person's age.
job	Categorical	Type of job (e.g., admin, services, etc.).
marital	Categorical	Marital status (e.g., divorced, married, etc.).
education	Categorical	Educational level (e.g., high.school, university.degree, etc.).
default	Binary	Has credit in default?
balance	Integer	Average yearly balance.
housing	Binary	Has housing loan?
loan	Binary	Has personal loan?
contact	Categorical	Contact communication type (e.g., cellular, telephone).
day_of_week	Date	Day of the week when the last contact was made.
month	Date	The month when the last contact was made.
duration	Integer	The duration, in seconds, of the last contact made.
campaign	Integer	Number of contacts performed during this campaing for this specific client, including the last contact.
pdays	Integer	Number of days that have passed by after the client was last contacted from a previous campaign (-1 means client was not previously contacted).
previous	Integer	Number of contacts performed for this client before this campaign.
poutcome	Categorical	Outcome of the previous marketing campaign (success, failure, etc.)
y	Binary	Target variable. It answers if the client has subscribed to a term deposit.

# Importing libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.patches as mpatches
import plotly.graph_objects as go
from plotly.subplots import make_subplots
import plotly.figure_factory as ff
import plotly.express as px
import time
import optuna
import joblib
import os
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate, cross_val_score
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, PowerTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              StackingClassifier, VotingClassifier, HistGradientBoostingClassifier)
from lightgbm import LGBMClassifier
from xgboost import XGBClassifier
from sklearn.metrics import (confusion_matrix, precision_score, recall_score,
                             f1_score, roc_auc_score, RocCurveDisplay, classification_report,
                             roc_curve, auc)
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline
import warnings
warnings.filterwarnings('ignore')
import plotly.io as pio
pio.renderers.default = 'plotly_mimetype+notebook'

# Settings and configurations
sns.set_style('whitegrid')
pd.set_option('display.max_columns', None)

# Defining seed for reproducibility
seed = 42

---

---

Exploratory Data Analysis

In this section, we'll conduct an EDA to understand the dataset. Our goal is to assess data quality, check missing values, and understand the relationship between features and the target variable. The insights gained here will be critical for guiding our feature engineering and modeling strategies in the next stages.

# Defining functions for EDA
def display_feature_list(features, feature_type):
    '''
    This function displays the features within each list for each type of data
    '''

    print(f"\n{feature_type} Features: ")
    print(', '.join(features) if features else 'None')


def describe_df(df):
    """
    This function prints some basic info on the dataset and 
    sets global variables for feature lists.
    """

    global categorical_features, continuous_features, binary_features
    categorical_features = [
        col for col in df.columns if not pd.api.types.is_numeric_dtype(df[col])
    ]
    
    binary_features = [col for col in df.columns if df[col].nunique(
    ) <= 2 and df[col].dtype != 'object']
    continuous_features = [
        col for col in df.columns if pd.api.types.is_numeric_dtype(df[col])
    ]

    print(f"\n{type(df).__name__} shape: {df.shape}")
    print(f"\n{df.shape[0]:,.0f} samples")
    print(f"\n{df.shape[1]:,.0f} attributes")
    display_feature_list(categorical_features, 'Categorical')
    display_feature_list(continuous_features, 'Continuous')
    display_feature_list(binary_features, 'Binary (0 or 1)')
    print(f'\nData Types: \n{df.dtypes}')
    print(
        f'\nMissing Data Percentage: \n{(df.isnull().sum() / len(df) * 100).round(2).astype(str) + "%"}')
    print(f'\nDuplicates: {df.duplicated().sum()}')
    print(
        f'\nCategorical Feature Cardinality: \n{df[categorical_features].nunique().sort_values(ascending=False)}')
    print(
        f'\nFeatures with Zero Variance: {", ".join([col for col in df.columns if df[col].nunique() == 1]) if any(df[col].nunique() == 1 for col in df.columns) else "None"}')

    negative_valued_features = [
        col for col in df[continuous_features] if (df[col] < 0).any()]
    print(
        f'\nFeatures with Negative Values: {", ".join(negative_valued_features) if negative_valued_features else "None"}')

    print('\nStatistical Summary: \n')
    display(df.describe().T)
    print(f'\n{type(df).__name__} Head: \n')
    display(df.head(5))
    print(f'\n{type(df).__name__} Tail: \n')
    display(df.tail(5))

# Loading the dataset
df = pd.read_csv('../data/raw/bank_marketing.csv')
describe_df(df)

DataFrame shape: (45211, 17)

45,211 samples

17 attributes

Categorical Features: 
job, marital, education, default, housing, loan, contact, month, poutcome, y

Continuous Features: 
age, balance, day_of_week, duration, campaign, pdays, previous

Binary (0 or 1) Features: 
None

Data Types: 
age             int64
job            object
marital        object
education      object
default        object
balance         int64
housing        object
loan           object
contact        object
day_of_week     int64
month          object
duration        int64
campaign        int64
pdays           int64
previous        int64
poutcome       object
y              object
dtype: object

Missing Data Percentage: 
age              0.0%
job             0.64%
marital          0.0%
education       4.11%
default          0.0%
balance          0.0%
housing          0.0%
loan             0.0%
contact         28.8%
day_of_week      0.0%
month            0.0%
duration         0.0%
campaign         0.0%
pdays            0.0%
previous         0.0%
poutcome       81.75%
y                0.0%
dtype: object

Duplicates: 0

Categorical Feature Cardinality: 
month        12
job          11
marital       3
education     3
poutcome      3
default       2
housing       2
loan          2
contact       2
y             2
dtype: int64

Features with Zero Variance: None

Features with Negative Values: balance, pdays

Statistical Summary: 

	count	mean	std	min	25%	50%	75%	max
age	45211.0	40.936210	10.618762	18.0	33.0	39.0	48.0	95.0
balance	45211.0	1362.272058	3044.765829	-8019.0	72.0	448.0	1428.0	102127.0
day_of_week	45211.0	15.806419	8.322476	1.0	8.0	16.0	21.0	31.0
duration	45211.0	258.163080	257.527812	0.0	103.0	180.0	319.0	4918.0
campaign	45211.0	2.763841	3.098021	1.0	1.0	2.0	3.0	63.0
pdays	45211.0	40.197828	100.128746	-1.0	-1.0	-1.0	-1.0	871.0
previous	45211.0	0.580323	2.303441	0.0	0.0	0.0	0.0	275.0

DataFrame Head: 

	age	job	marital	education	default	balance	housing	loan	contact	day_of_week	month	duration	campaign	pdays	previous	poutcome	y
0	58	management	married	tertiary	no	2143	yes	no	NaN	5	may	261	1	-1	0	NaN	no
1	44	technician	single	secondary	no	29	yes	no	NaN	5	may	151	1	-1	0	NaN	no
2	33	entrepreneur	married	secondary	no	2	yes	yes	NaN	5	may	76	1	-1	0	NaN	no
3	47	blue-collar	married	NaN	no	1506	yes	no	NaN	5	may	92	1	-1	0	NaN	no
4	33	NaN	single	NaN	no	1	no	no	NaN	5	may	198	1	-1	0	NaN	no

DataFrame Tail: 

	age	job	marital	education	default	balance	housing	loan	contact	day_of_week	month	duration	campaign	pdays	previous	poutcome	y
45206	51	technician	married	tertiary	no	825	no	no	cellular	17	nov	977	3	-1	0	NaN	yes
45207	71	retired	divorced	primary	no	1729	no	no	cellular	17	nov	456	2	-1	0	NaN	yes
45208	72	retired	married	secondary	no	5715	no	no	cellular	17	nov	1127	5	184	3	success	yes
45209	57	blue-collar	married	secondary	no	668	no	no	telephone	17	nov	508	4	-1	0	NaN	no
45210	37	entrepreneur	married	secondary	no	2971	no	no	cellular	17	nov	361	2	188	11	other	no

• Missing Data: The dataset contains significant missing values that we must investigate further.
  • The poutcome feature is the most extreme case, with over 80% of its data missing.
  • The variables contact, education, and job also have a notable number of nulls that will require attention.
• Feature Interpretation
  • We have reasons to believe that the column day_of_week is mislabeled, given it ranges from 1 to 31, signaling it refers to the day of the month, instead of day of the week.
  • We can see that many numerical features, such as balance, campaign, and duration, are heavily right-skewed, indicating that transformations will be necessary before feeding them to our baseline model. That's because linear models, like Logistic Regression, are very sensitive to the scale of the data and can become biased towards values that are far away from the rest of the data.
• Data Leakage
  • Feature duration represents a data leak. Its value is unknown before a call is made, so it must be dropped from our dataset before building our models.

Categorical Features

Our data has a set of both categorical and continuous features. Let's first focus on the categorical features. We will start by analyzing the distribution of the target variable y and then see how the subscription rate varies across different categories.

# EDA Plot
def plot_stacked_bar_percentage(df, feat, target, positive_class='yes'):
    """
    This function plots a 100% stacked bar chart using Plotly to show the relationship
    between a categorical feature and a binary target variable.
    """
    feature_data = df[feat].fillna('Missing')
    ct = pd.crosstab(feature_data, df[target], normalize='index')
    ct = ct.sort_values(by=positive_class, ascending=True)

    palette = {'no': '#29788E', 'yes': '#E85B5B'}

    fig = go.Figure()

    for col in ct.columns:
        fig.add_trace(go.Bar(
            y=ct.index,
            x=ct[col],
            name=col,
            orientation='h',
            marker_color=palette.get(col, '#cccccc'),
            text=ct[col].apply(lambda x: f'{x:.1%}'),
            textposition='inside',
            insidetextfont=dict(color='white', size=12)
        ))

    fig.update_layout(
        barmode='stack',
        title={
            'text': f"<b>Proportion of '{target.title()}' by '{feat.replace('_',' ').title()}'</b>",
            'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
        },
        xaxis=dict(
            title='Proportion',
            tickformat='.0%'
        ),
        yaxis=dict(
            title=feat.replace('_',' ').title()
        ),
        legend_title_text=target.title(),
        plot_bgcolor='#f5f7f6',
        paper_bgcolor='#f5f7f6',
        height=600,
        margin=dict(t=50, l=120)
    )

    fig.show()

# EDA Plots
def plot_donut(df, feat):
    """
    This function plots a donut chart to show the distribution of a categorical feature.
    """
    counts = df[feat].value_counts()
    labels = counts.index
    values = counts.values
    total = df[feat].count()

    palette = {
        'no': '#29788E',
        'yes': '#E85B5B'
    }
    colors = [palette.get(label, '#CCCCCC') for label in labels]

    fig = go.Figure(data=[go.Pie(
        labels=labels,
        values=values,
        hole=0.7,
        marker_colors=colors,
        textinfo='percent',
        hoverinfo='label+percent+value',
        insidetextfont=dict(color='white', size=12, family='Arial, bold')
    )])

    fig.update_layout(
        title={
            'text': f"<b>Proportion of Target Variable: '{feat.title()}'</b>",
            'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
        },
        annotations=[dict(
            text=f'Total:<br>{total:,}',
            x=0.5, y=0.5,
            font_size=20,
            showarrow=False
        )],
        legend=dict(
            title=feat.title(),
            yanchor="top", y=0.9, xanchor="left", x=0.9
        ),
        plot_bgcolor='#f5f7f6',
        paper_bgcolor='#f5f7f6',
        height=500,
        margin=dict(t=50, b=20, l=20, r=20)
    )

    fig.show()

# Obtaining the proportion of labels in the target variable
plot_donut(df, 'y')

• Class Distribution
  • The chart above shows that our dataset is highly imbalanced, meaning that both classes are not represented equally. The negative class no is the overwhelming majority, accounting for 88.3% (39,922 instances) of a total of 45,211 samples. The yes class, on the other hand, accounts for a small minority of 5,289 instances (11.7% of the total).
    
    
  • Given this context, a lazy model could achieve 88.3% accuracy by simply predicting 'no' every single time. Accuracy will be a misleading metric.
    
    
• Dealing with Imblanced Labels
  • Considering the class imbalance, we must focus on other metrics besides accuracy. We must also use the right strategy to deal with imbalance, such as defining the class_weight='balanced' param in our models or use techniques like SMOTE to oversample the minority class.

# Observing the relationship between categorical features and the target variable
categorical_features.remove('y')
for feat in df[categorical_features]:
    plot_stacked_bar_percentage(df, feat, 'y')

• Time-Based & Behavioral Features
  • The month variable appears to be a good predictor. There is a clear seasonal pattern, where campaings in March, September, October and December are wildly successful. May and July, on the other hand, presented worse outcomes.
    
    
  • The previous campaign outcome (poutcome) is also a powerful indicator. Clients with a prior 'success' are more likely to subscribe again.
• Demographics & Occupation
  • The job type reveals interesting patterns as well. Those labeled as student and retired appears to have the highest propensity to subscribe. On the other hand, blue-collar workers and entrepreneur were the least likely.
    
    
  • A similar trend can be observed for marital status and education. Single clients and those with tertiary (i.e., higher education) had slightly higher success rates.
• Financial Status
  • Binary categorical features (housing, loan, and default) also show interesting insights. Subscriptions are more common proportionally among those with no existing housing loan, no personal loan, and no credit default. This could indicate that clients with fewer financial liabilities are a more receptive audience for this product.
    
    

Continuous Features

Now we'll turn our attention to the continuous features. We will analyze their distributions and explore how they relate to the subscription outcome. We will also use a correlation matrix to assess any relationships between the numerical features themselves.

# EDA Plots
def plot_hist_box(df, feat):
    """
    This function plots a histogram and a box plot side-by-side
    for a given continuous feature.
    """
    fig = make_subplots(
        rows=1, cols=2,
        column_widths=[0.2, 0.8],
        subplot_titles=("Box Plot", "Histogram")
    )

    fig.add_trace(
        go.Box(y=df[feat], name='', marker_color='#D32F2F', boxmean=True),
        row=1, col=1
    )

    fig.add_trace(
        go.Histogram(x=df[feat], name='', marker_color='#1976D2'),
        row=1, col=2
    )

    fig.update_layout(
        title={
            'text': f"<b>Distribution of {feat.replace('_',' ').title()}</b>",
            'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
        },
        showlegend=False,
        plot_bgcolor='#f5f7f6',
        paper_bgcolor='#f5f7f6',
        height=500,
        margin=dict(t=80)
    )

    fig.update_yaxes(title_text=feat.title(), row=1, col=1)
    fig.update_yaxes(title_text="Frequency", row=1, col=2)
    fig.update_xaxes(title_text=feat.title(), row=1, col=2)

    fig.show()

# EDA plots
def plot_kde_by_target(df, cont_feat, target):
    """
    This function plots overlapping KDEs for a continuous feature, 
    separated by the binary target variable.
    """
    palette = {
        'no': '#29788E',
        'yes': '#E85B5B'
    }

    group_labels = df[target].unique()
    hist_data = [df[df[target] == label][cont_feat] for label in group_labels]
    colors = [palette.get(label) for label in group_labels]

    fig = ff.create_distplot(
        hist_data,
        group_labels,
        show_hist=False,
        show_rug=False,
        colors=colors
    )
    
    for trace in fig.data:
        trace.update(fill='tozeroy', mode='lines', line=dict(width=2.5))
    
    fig.update_layout(
        title={
            'text': f"<b>Distribution of '{cont_feat.replace('_',' ').title()}' by '{target.title()}'</b>",
            'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
        },
        xaxis_title=cont_feat.replace('_',' ').title(),
        yaxis_title='Density',
        plot_bgcolor='#f5f7f6',
        paper_bgcolor='#f5f7f6',
        legend=dict(title=target.title()),
        height=500
    )
    
    fig.show()

# EDA Plots
def plot_correlation_heatmap(df):
    """
    This function plots a correlation heatmap for numerical features.
    """
    corr = df.corr(numeric_only=True).round(2)
    mask = np.triu(np.ones_like(corr, dtype=bool))
    corr_masked = corr.mask(mask)

    text_labels = corr_masked.applymap(lambda x: f'{x:.2f}' if pd.notna(x) else '')

    fig = go.Figure(data=go.Heatmap(
        z=corr_masked.values,
        x=corr_masked.columns,
        y=corr_masked.index,
        colorscale='YlOrBr',
        zmin=-1,
        zmax=1,
        text=text_labels.values,
        texttemplate="%{text}",
        textfont={"size":10}
    ))

    fig.update_layout(
        title={
            'text': '<b>Feature Correlation Heatmap</b>',
            'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
        },
        plot_bgcolor='#f5f7f6',
        paper_bgcolor='#f5f7f6',
        width=800,
        height=800,
        xaxis_showgrid=False,
        yaxis_showgrid=False,
        yaxis_autorange='reversed',
        yaxis_scaleanchor="x"
    )

    fig.update_xaxes(tickangle=-45)

    fig.show()

# Plotting Box Plot and Histogram for continuous features
for feat in continuous_features:
    plot_hist_box(df, feat)

• Skewness
  • The variables balance, duration, and campaign are heavily right-skewed. That means there is a much larger concentration of data points at the lower end with a long tail of extreme, but less frequent, values. For linear models (e.g., Support Vector Machines, Logistic Regression) and distance-based algorithms (e.g., KMeans), this skew can be problematic.
    
    
• Quasi-Categorical Features
  • The pdays and previous features are not truly continuous but behave as quasi-categorical variables. Each is dominated by a single value (-1 for pdays and 0 for previous), which accounts for over 80% of the data. This indicates they should be re-engineered into binary or categorical features rather than being treated as numerical.

# Plotting KDE plots for continuous features by target variable
for feat in continuous_features:
    plot_kde_by_target(df, feat, 'y')

• Data Leakage

  • The duration of the call is highly correlated with the outcome, with longer calls strongly indicating a subscription. However, we known this is a data leak, since the duration is only known after the call is made. We will remove this feature from the dataset before modeling.

• Behavioral Predictors

  • The number of contacts (campaign) appears to be a relevant predictor. Successful outcomes are heavily concentrated in the first one or two contacts. The data clearly shows that repeated contacts lead to diminishing returns, with a high number of calls being strongly associated with a negative outcome.
    
    
  • Similarly, prior engagement (pdays) is a positive signal. While most clients have never been contacted before (pdays = -1), those who have been previously contacted show a higher propensity to subscribe.

• Demographic Insights

  • The age distribution confirms our findings from the job analysis. There is a higher likelihood of subscription among clients at the extremes of the age range: younger clients (typically students) and older clients (typically retired).

# Plotting correlation heatmap
plot_correlation_heatmap(df)

• Moderate Positive Correlation
  • The most significant correlation is a moderate positive relationship between pdays and previous. This is expected, because both features are related to past contact history. If a client has had previous contact (previous > 0), then pdays will also have a value other than -1. They are measuring a similar underlying concept.
    
    
• Absence of Linear Trends
  • Overall, the correlation heatmap shows that there are no linear trends among continuous features (e.g., "as age increases, balance increases). This lack of strong linear relationships suggest that linear models might struggle to find patterns without a significant level of feature engineering. Non-linear models, like Random Forest or Gradient Boosting, might be able to capture more complex relationships between features.

---

---

Data Pipeline & Preprocessing

In this section, we will start by building a baseline. We will then perform some feature engineering based on our findings and build the data pipeline that will be used to automate and encapsulate all our preprocessing workflow.

Baseline

Let's start by building a baseline with a simple Logistic Regression model to evaluate performance.

During our EDA, we uncovered that poutcome feature has over 80% of missing data. Our first inclination was to drop this feature, due to a high number of missing data. However, given that this feature appears to have important predictive power (64.7% of clients whose previous marketing campaign was a success said yes to this current campaign), we might want to test performance of a baseline model with and without this feature.

If including poutcome gives us a signifcant performance boost, we will know it is worth it to keep this feature. However, if performance is worse, we will opt for excluding it.

# Creating a copy of the original dataframe for preprocessing and modelling
df_model = df.copy()

# Dropping the 'duration' feature due to data leakage
df_model.drop(columns=['duration'], axis=1, inplace=True)

# Filling missing values in categorical features with 'Unknown'
df_model[categorical_features] = df_model[categorical_features].fillna('unknown')

# Separating features and target variable
X = df_model.drop('y', axis=1)
y = df_model['y'].map({'yes': 1, 'no': 0}) # Encoding target variable as binary

print("--- Features and Target Variable ---")
print("\nFeatures:\n")
print(X.shape)
print('\n')
print(X.head())
print("\nTarget Variable:\n")
print(y.shape)
print('\n')
print(y.head())
print("\n------------------------------------\n")
print("Missing Data Check:\n")
print(df_model.isnull().sum())
print("\n------------------------------------\n")

--- Features and Target Variable ---

Features:

(45211, 15)


   age           job  marital  education default  balance housing loan  \
0   58    management  married   tertiary      no     2143     yes   no   
1   44    technician   single  secondary      no       29     yes   no   
2   33  entrepreneur  married  secondary      no        2     yes  yes   
3   47   blue-collar  married    unknown      no     1506     yes   no   
4   33       unknown   single    unknown      no        1      no   no   

   contact  day_of_week month  campaign  pdays  previous poutcome  
0  unknown            5   may         1     -1         0  unknown  
1  unknown            5   may         1     -1         0  unknown  
2  unknown            5   may         1     -1         0  unknown  
3  unknown            5   may         1     -1         0  unknown  
4  unknown            5   may         1     -1         0  unknown  

Target Variable:

(45211,)


0    0
1    0
2    0
3    0
4    0
Name: y, dtype: int64

------------------------------------

Missing Data Check:

age            0
job            0
marital        0
education      0
default        0
balance        0
housing        0
loan           0
contact        0
day_of_week    0
month          0
campaign       0
pdays          0
previous       0
poutcome       0
y              0
dtype: int64

------------------------------------

# Splitting the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=seed, stratify=y)
print("--- Train-Test Split ---\n")
print(f"Training set shape: {X_train.shape}, {y_train.shape}")
print(f"Testing set shape: {X_test.shape}, {y_test.shape}")

--- Train-Test Split ---

Training set shape: (36168, 15), (36168,)
Testing set shape: (9043, 15), (9043,)

# Evaluation function for model performance
def plot_confusion_matrix(y_true, y_pred, model_name):
    """
    This function plots a confusion matrix for the given true and predicted labels.
    """
    cm = confusion_matrix(y_true, y_pred)
    labels = ['No (0)', 'Yes (1)']

    fig = ff.create_annotated_heatmap(
        z=cm,
        x=labels,
        y=labels,
        colorscale='Blues',
        showscale=False
    )

    fig.update_layout(
        title={
            'text': f'<b>Confusion Matrix for {model_name}</b>',
            'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
        },
        plot_bgcolor='#f5f7f6',
        paper_bgcolor='#f5f7f6',
        width=500,
        height=500
    )

    fig.update_xaxes(title_text='Predicted Label')
    fig.update_yaxes(title_text='True Label', autorange='reversed')
    
    for i in range(len(fig.layout.annotations)):
        fig.layout.annotations[i].font.size = 16

    fig.show()

# --- Experimenting Baseline Model Without `poutcome` ---
# Dropping `poutcome` feature
X_train_no_poutcome = X_train.drop(columns=['poutcome'], axis=1)
X_test_no_poutcome = X_test.drop(columns=['poutcome'], axis=1)

# Identifying categorical and numerical features for preprocessing
categorical_features_no_poutcome = X_train_no_poutcome.select_dtypes(include=['object']).columns
numerical_features_no_poutcome = X_train_no_poutcome.select_dtypes(include=np.number).columns
print("\nCategorical Features (without 'poutcome'):\n", categorical_features_no_poutcome)
print("\nNumerical Features (without 'poutcome'):\n", numerical_features_no_poutcome)
print("\n------------------------------------\n")

# Creating preprocessor for encoding categorical features and scaling numerical features
preprocessor_no_poutcome = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features_no_poutcome),
        ('num', PowerTransformer(method='yeo-johnson'), numerical_features_no_poutcome)
    ])
print("\nPreprocessor (without 'poutcome'):\n", preprocessor_no_poutcome)
print("\n------------------------------------\n")

# Creating a pipeline 
pipeline_no_poutcome = Pipeline(steps=[
    ('preprocessor', preprocessor_no_poutcome),
    ('classifier', LogisticRegression(class_weight='balanced', random_state=seed, max_iter=1000))
])
print("\nPipeline (without 'poutcome'):\n", pipeline_no_poutcome)
print("\n------------------------------------\n")

# Training and evaluating the model without `poutcome`
pipeline_no_poutcome.fit(X_train_no_poutcome, y_train)
y_pred_no_poutcome = pipeline_no_poutcome.predict(X_test_no_poutcome)
y_proba_no_poutcome = pipeline_no_poutcome.predict_proba(X_test_no_poutcome)[:, 1]

# --- Experimenting Baseline Model With `poutcome` ---
# Identifying categorical and numerical features for preprocessing
categorical_features_with_poutcome = X_train.select_dtypes(include=['object']).columns
numerical_features_with_poutcome = X_train.select_dtypes(include=np.number).columns
print("\nCategorical Features (with 'poutcome'):\n", categorical_features_with_poutcome)
print("\nNumerical Features (with 'poutcome'):\n", numerical_features_with_poutcome)
print("\n------------------------------------\n")

# Creating preprocessor for encoding categorical features and scaling numerical features
preprocessor_with_poutcome = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features_with_poutcome),
        ('num', PowerTransformer(method='yeo-johnson'), numerical_features_with_poutcome)
    ])
print("\nPreprocessor (with 'poutcome'):\n", preprocessor_with_poutcome)
print("\n------------------------------------\n")

# Creating a pipeline 
pipeline_with_poutcome = Pipeline(steps=[
    ('preprocessor', preprocessor_with_poutcome),
    ('classifier', LogisticRegression(class_weight='balanced', random_state=seed, max_iter=1000))
])
print("\nPipeline (with 'poutcome'):\n", pipeline_with_poutcome)
print("\n------------------------------------\n")

# Training and evaluating the model with `poutcome`
pipeline_with_poutcome.fit(X_train, y_train)
y_pred_with_poutcome = pipeline_with_poutcome.predict(X_test)
y_proba_with_poutcome = pipeline_with_poutcome.predict_proba(X_test)[:, 1]

# --- Performance Comparison ---
print("\n--- Model Performance Comparison ---\n")
print("\nClassification Report (without 'poutcome'):\n")
print(classification_report(y_test, y_pred_no_poutcome, target_names=['no', 'yes']))
print("\nClassification Report (with 'poutcome'):\n")
print(classification_report(y_test, y_pred_with_poutcome, target_names=['no', 'yes']))

# Plotting ROC Curves for both models
fpr_no_p, tpr_no_p, _ = roc_curve(y_test, y_proba_no_poutcome, pos_label=1)
auc_no_p = auc(fpr_no_p, tpr_no_p)
fpr_with_p, tpr_with_p, _ = roc_curve(y_test, y_proba_with_poutcome, pos_label=1)
auc_with_p = auc(fpr_with_p, tpr_with_p)

fig = go.Figure()
fig.add_trace(go.Scatter(
    x=fpr_no_p, 
    y=tpr_no_p,
    mode='lines',
    line=dict(color='#29788E', width=2),
    name=f"Without 'poutcome' (AUC = {auc_no_p:.2f})"
))
fig.add_trace(go.Scatter(
    x=fpr_with_p, 
    y=tpr_with_p,
    mode='lines',
    line=dict(color='#E85B5B', width=2),
    name=f"With 'poutcome' (AUC = {auc_with_p:.2f})"
))
fig.add_trace(go.Scatter(
    x=[0, 1], 
    y=[0, 1],
    mode='lines',
    line=dict(color='black', width=2, dash='dash'),
    name='Random Guess'
))
fig.update_layout(
    title='<b>ROC Curve Comparison</b>',
    xaxis_title='False Positive Rate',
    yaxis_title='True Positive Rate',
    xaxis=dict(range=[0.0, 1.0]),
    yaxis=dict(range=[0.0, 1.05]),
    width=700,
    height=700,
    plot_bgcolor='#f5f7f6',
    paper_bgcolor='#f5f7f6',
    legend=dict(x=0.55, y=0.1),
    xaxis_scaleanchor="y",
    yaxis_scaleratio=1
)
fig.show()

# Plotting Confusion Matrices for both models
plot_confusion_matrix(y_test, y_pred_no_poutcome, "Without 'poutcome'")
plot_confusion_matrix(y_test, y_pred_with_poutcome, "With 'poutcome'")

# Building a Comparison Dataframe
metrics = {
    'Model': ["Without 'poutcome'", "With 'poutcome'"],
    'Precision': [
        precision_score(y_test, y_pred_no_poutcome),
        precision_score(y_test, y_pred_with_poutcome)
    ],
    'Recall': [
        recall_score(y_test, y_pred_no_poutcome),
        recall_score(y_test, y_pred_with_poutcome)
    ],
    'F1-Score': [
        f1_score(y_test, y_pred_no_poutcome),
        f1_score(y_test, y_pred_with_poutcome)
    ],
    'ROC AUC': [
        roc_auc_score(y_test, y_proba_no_poutcome),
        roc_auc_score(y_test, y_proba_with_poutcome)
    ]
}
comparison_df = pd.DataFrame(metrics)
print("\nPerformance Comparison DataFrame:\n")
display(comparison_df.style.format({'Precision': '{:.2f}', 'Recall': '{:.2f}', 'F1-Score': '{:.2f}', 'ROC AUC': '{:.2f}'}))

Categorical Features (without 'poutcome'):
 Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'month'],
      dtype='object')

Numerical Features (without 'poutcome'):
 Index(['age', 'balance', 'day_of_week', 'campaign', 'pdays', 'previous'], dtype='object')

------------------------------------


Preprocessor (without 'poutcome'):
 ColumnTransformer(transformers=[('cat', OneHotEncoder(handle_unknown='ignore'),
                                 Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'month'],
      dtype='object')),
                                ('num', PowerTransformer(),
                                 Index(['age', 'balance', 'day_of_week', 'campaign', 'pdays', 'previous'], dtype='object'))])

------------------------------------


Pipeline (without 'poutcome'):
 Pipeline(steps=[('preprocessor',
                 ColumnTransformer(transformers=[('cat',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'month'],
      dtype='object')),
                                                 ('num', PowerTransformer(),
                                                  Index(['age', 'balance', 'day_of_week', 'campaign', 'pdays', 'previous'], dtype='object'))])),
                ('classifier',
                 LogisticRegression(class_weight='balanced', max_iter=1000,
                                    random_state=42))])

------------------------------------


Categorical Features (with 'poutcome'):
 Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'month', 'poutcome'],
      dtype='object')

Numerical Features (with 'poutcome'):
 Index(['age', 'balance', 'day_of_week', 'campaign', 'pdays', 'previous'], dtype='object')

------------------------------------


Preprocessor (with 'poutcome'):
 ColumnTransformer(transformers=[('cat', OneHotEncoder(handle_unknown='ignore'),
                                 Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'month', 'poutcome'],
      dtype='object')),
                                ('num', PowerTransformer(),
                                 Index(['age', 'balance', 'day_of_week', 'campaign', 'pdays', 'previous'], dtype='object'))])

------------------------------------


Pipeline (with 'poutcome'):
 Pipeline(steps=[('preprocessor',
                 ColumnTransformer(transformers=[('cat',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'month', 'poutcome'],
      dtype='object')),
                                                 ('num', PowerTransformer(),
                                                  Index(['age', 'balance', 'day_of_week', 'campaign', 'pdays', 'previous'], dtype='object'))])),
                ('classifier',
                 LogisticRegression(class_weight='balanced', max_iter=1000,
                                    random_state=42))])

------------------------------------


--- Model Performance Comparison ---


Classification Report (without 'poutcome'):

              precision    recall  f1-score   support

          no       0.95      0.72      0.82      7985
         yes       0.25      0.69      0.36      1058

    accuracy                           0.72      9043
   macro avg       0.60      0.70      0.59      9043
weighted avg       0.86      0.72      0.76      9043


Classification Report (with 'poutcome'):

              precision    recall  f1-score   support

          no       0.94      0.77      0.85      7985
         yes       0.27      0.64      0.38      1058

    accuracy                           0.76      9043
   macro avg       0.61      0.71      0.62      9043
weighted avg       0.86      0.76      0.79      9043

Performance Comparison DataFrame:

	Model	Precision	Recall	F1-Score	ROC AUC
0	Without 'poutcome'	0.25	0.69	0.36	0.76
1	With 'poutcome'	0.27	0.64	0.38	0.77

• Performance Evaluation

  • Although the improvement is not massive, the model that includes the poutcome feature demonstrated superior performance on our key metrics. It achieved a higher F1-Score (0.38 vs. 0.35), which indicates a better balance between Precision and Recall. While its Recall was slightly lower, its Precision was higher, resulting in significantly fewer False Positive errors (1,804 vs. 2,239).

• Business POV

  • From a business perspective, an effective marketing campaign must balance two objectives: maximizing the number of successful sales (Recall) while minimizing resources spent on uninterested clients (Precision). The F1-Score is the ideal metric for this scenario as it measures the overall effectiveness of the model's generated call list by finding a harmony between these two goals. Since the model with poutcome achieved a higher F1-Score, it provides more business value. Therefore, we will keep this feature for our final model.

Feature Engineering

We will maintain the poutcome feature in the dataset. In this section, we will fill in missing data with the unknown label across categories. We will also drop the duration feature due to data leakage and re-engineer quasi-categorical features.

# Creating a copy of the model dataframe for feature engineering
df_engineered = df.copy()

# Dropping the `duration` feature due to data leakage
df_engineered.drop(columns=['duration'], axis=1, inplace=True)

# Replacing missing values with `unknown`
df_engineered[categorical_features] = df_engineered[categorical_features].fillna('unknown')

print("\n--- Initial Cleaning Complete ---\n")
print("Feature `duration` dropped and NaNs filled with 'unknown'.\n")
print("Original Dataframe:\n")
display(df.head(5))
print("\nEngineered Dataframe:\n")
display(df_engineered.head(5))
print("\n------------------------------------\n")

--- Initial Cleaning Complete ---

Feature `duration` dropped and NaNs filled with 'unknown'.

Original Dataframe:

	age	job	marital	education	default	balance	housing	loan	contact	day_of_week	month	duration	campaign	pdays	previous	poutcome	y
0	58	management	married	tertiary	no	2143	yes	no	NaN	5	may	261	1	-1	0	NaN	no
1	44	technician	single	secondary	no	29	yes	no	NaN	5	may	151	1	-1	0	NaN	no
2	33	entrepreneur	married	secondary	no	2	yes	yes	NaN	5	may	76	1	-1	0	NaN	no
3	47	blue-collar	married	NaN	no	1506	yes	no	NaN	5	may	92	1	-1	0	NaN	no
4	33	NaN	single	NaN	no	1	no	no	NaN	5	may	198	1	-1	0	NaN	no

Engineered Dataframe:

	age	job	marital	education	default	balance	housing	loan	contact	day_of_week	month	campaign	pdays	previous	poutcome	y
0	58	management	married	tertiary	no	2143	yes	no	unknown	5	may	1	-1	0	unknown	no
1	44	technician	single	secondary	no	29	yes	no	unknown	5	may	1	-1	0	unknown	no
2	33	entrepreneur	married	secondary	no	2	yes	yes	unknown	5	may	1	-1	0	unknown	no
3	47	blue-collar	married	unknown	no	1506	yes	no	unknown	5	may	1	-1	0	unknown	no
4	33	unknown	single	unknown	no	1	no	no	unknown	5	may	1	-1	0	unknown	no

------------------------------------

• EDA Insights
  • During EDA, we observed that pdays and previous features have an overwhelming number of instances at the same value (-1 and 0). In the first run of this notebook, we transformed these features into binary variables.
    
    
  • Essentially, instead of pdays, we created a new feature called was_previously_contacted where values other than -1 were labeled yes.
    
    
  • Likewise, we transformed the previous feature into a feature called had_previous_contact, where values other than 0 were labeled yes.
    
    
  • In this run, we will maintain these variables and keep them as numerical values, passing them through the PowerTransformer in the preprocessor.

• Dealing with Mislabeled Features
  • Our EDA also revealed that the feature day_of_week is misleading, as it clearly refers to the day of the month.
    
    
  • In our initial experiments, we renamed this feature to day_of_month to reflect its nature better. We changed it into a categorical feature.
    
    
  • In this run, we will apply cyclical encoding to the day_of_month and month features. This technique addresses the fact that time-based features are cyclical (e.g., December is right next to January). By converting these features into sine and cosine components, we map them onto a 2D circle, preserving this natural continuity for the model.

# Encoding months 
month_map = {
    'jan': 1, 'feb': 2, 'mar': 3, 'apr': 4,
    'may': 5, 'jun': 6, 'jul': 7, 'aug': 8,
    'sep': 9, 'oct': 10, 'nov': 11, 'dec': 12
}
df_engineered['month_num'] = df_engineered['month'].map(month_map)

# Cyclical encoding for `month` and `day_of_week`
df_engineered['month_sin'] = np.sin(2 * np.pi * df_engineered['month_num'] / 12)
df_engineered['month_cos'] = np.cos(2 * np.pi * df_engineered['month_num'] / 12)

df_engineered['day_sin'] = np.sin(2 * np.pi * df_engineered['day_of_week'] / 31)
df_engineered['day_cos'] = np.cos(2 * np.pi * df_engineered['day_of_week'] / 31)

# Dropping original `month` and `day_of_week` columns
df_engineered = df_engineered.drop(columns=['month', 'month_num', 'day_of_week'])

# Check result
print(df_engineered[['month_sin','month_cos','day_sin','day_cos']].head())

   month_sin  month_cos   day_sin   day_cos
0        0.5  -0.866025  0.848644  0.528964
1        0.5  -0.866025  0.848644  0.528964
2        0.5  -0.866025  0.848644  0.528964
3        0.5  -0.866025  0.848644  0.528964
4        0.5  -0.866025  0.848644  0.528964

Before moving on to the next phase, we can run a quick sanity check by using our describe_df function to see if the post-engineered dataframe is how we initially planned.

# Checking the engineered dataframe
describe_df(df_engineered)

DataFrame shape: (45211, 18)

45,211 samples

18 attributes

Categorical Features: 
job, marital, education, default, housing, loan, contact, poutcome, y

Continuous Features: 
age, balance, campaign, pdays, previous, month_sin, month_cos, day_sin, day_cos

Binary (0 or 1) Features: 
None

Data Types: 
age            int64
job           object
marital       object
education     object
default       object
balance        int64
housing       object
loan          object
contact       object
campaign       int64
pdays          int64
previous       int64
poutcome      object
y             object
month_sin    float64
month_cos    float64
day_sin      float64
day_cos      float64
dtype: object

Missing Data Percentage: 
age          0.0%
job          0.0%
marital      0.0%
education    0.0%
default      0.0%
balance      0.0%
housing      0.0%
loan         0.0%
contact      0.0%
campaign     0.0%
pdays        0.0%
previous     0.0%
poutcome     0.0%
y            0.0%
month_sin    0.0%
month_cos    0.0%
day_sin      0.0%
day_cos      0.0%
dtype: object

Duplicates: 16

Categorical Feature Cardinality: 
job          12
education     4
poutcome      4
marital       3
contact       3
default       2
housing       2
loan          2
y             2
dtype: int64

Features with Zero Variance: None

Features with Negative Values: balance, pdays, month_sin, month_cos, day_sin, day_cos

Statistical Summary: 

	count	mean	std	min	25%	50%	75%	max
age	45211.0	40.936210	10.618762	18.000000	33.000000	3.900000e+01	48.000000	95.000000
balance	45211.0	1362.272058	3044.765829	-8019.000000	72.000000	4.480000e+02	1428.000000	102127.000000
campaign	45211.0	2.763841	3.098021	1.000000	1.000000	2.000000e+00	3.000000	63.000000
pdays	45211.0	40.197828	100.128746	-1.000000	-1.000000	-1.000000e+00	-1.000000	871.000000
previous	45211.0	0.580323	2.303441	0.000000	0.000000	0.000000e+00	0.000000	275.000000
month_sin	45211.0	0.018449	0.612731	-1.000000	-0.500000	1.224647e-16	0.500000	1.000000
month_cos	45211.0	-0.470301	0.634865	-1.000000	-0.866025	-8.660254e-01	-0.500000	1.000000
day_sin	45211.0	0.027214	0.698901	-0.998717	-0.651372	-1.011683e-01	0.724793	0.998717
day_cos	45211.0	-0.106996	0.706661	-0.994869	-0.758758	-2.506525e-01	0.528964	1.000000

DataFrame Head: 

	age	job	marital	education	default	balance	housing	loan	contact	campaign	pdays	previous	poutcome	y	month_sin	month_cos	day_sin	day_cos
0	58	management	married	tertiary	no	2143	yes	no	unknown	1	-1	0	unknown	no	0.5	-0.866025	0.848644	0.528964
1	44	technician	single	secondary	no	29	yes	no	unknown	1	-1	0	unknown	no	0.5	-0.866025	0.848644	0.528964
2	33	entrepreneur	married	secondary	no	2	yes	yes	unknown	1	-1	0	unknown	no	0.5	-0.866025	0.848644	0.528964
3	47	blue-collar	married	unknown	no	1506	yes	no	unknown	1	-1	0	unknown	no	0.5	-0.866025	0.848644	0.528964
4	33	unknown	single	unknown	no	1	no	no	unknown	1	-1	0	unknown	no	0.5	-0.866025	0.848644	0.528964

DataFrame Tail: 

	age	job	marital	education	default	balance	housing	loan	contact	campaign	pdays	previous	poutcome	y	month_sin	month_cos	day_sin	day_cos
45206	51	technician	married	tertiary	no	825	no	no	cellular	3	-1	0	unknown	yes	-0.5	0.866025	-0.299363	-0.954139
45207	71	retired	divorced	primary	no	1729	no	no	cellular	2	-1	0	unknown	yes	-0.5	0.866025	-0.299363	-0.954139
45208	72	retired	married	secondary	no	5715	no	no	cellular	5	184	3	success	yes	-0.5	0.866025	-0.299363	-0.954139
45209	57	blue-collar	married	secondary	no	668	no	no	telephone	4	-1	0	unknown	no	-0.5	0.866025	-0.299363	-0.954139
45210	37	entrepreneur	married	secondary	no	2971	no	no	cellular	2	188	11	other	no	-0.5	0.866025	-0.299363	-0.954139

We now have 16 duplicates. We will drop them down below.

You can also see that features month_sin, month_cos, day_sin, and day_cos all range from around -1 to 1. We don't need to apply PowerTransformer to these cyclical features and we will pass them through the distribution transformation.

# Dropping duplicates
df_engineered.drop_duplicates(inplace=True)
print(f"Duplicates after dropping: {df_engineered.duplicated().sum()}")
print(f"New shape after dropping duplicates: {df_engineered.shape}")
print("\n------------------------------------\n")

Duplicates after dropping: 0
New shape after dropping duplicates: (45195, 18)

------------------------------------

---

---

Modeling

With our data fully preprocessed and engineered, we now proceed to the modeling phase. In this section, we'll train and evaluate a variety of classification algorithms to find the best performer for our business problem. Each model will be evaluated using a stratified cross-validation strategy to ensure fair and robust comparison.

# Removing `y` from categorical_features list
categorical_features.remove('y')

# Separating features and target variable
X_engineered = df_engineered.drop('y', axis=1)
y_engineered = df_engineered['y'].map({'yes': 1, 'no': 0}) # Encoding target variable as binary

print("--- Features and Target Variable ---")
print("\nFeatures:\n")
print(X_engineered.shape)
print('\n')
print(X_engineered.head())
print("\nTarget Variable:\n")
print(y_engineered.shape)
print('\n')
print(y_engineered.head())
print("\n------------------------------------\n")
print("Missing Data Check:\n")
print(df_engineered.isnull().sum())
print("\n------------------------------------\n")

--- Features and Target Variable ---

Features:

(45195, 17)


   age           job  marital  education default  balance housing loan  \
0   58    management  married   tertiary      no     2143     yes   no   
1   44    technician   single  secondary      no       29     yes   no   
2   33  entrepreneur  married  secondary      no        2     yes  yes   
3   47   blue-collar  married    unknown      no     1506     yes   no   
4   33       unknown   single    unknown      no        1      no   no   

   contact  campaign  pdays  previous poutcome  month_sin  month_cos  \
0  unknown         1     -1         0  unknown        0.5  -0.866025   
1  unknown         1     -1         0  unknown        0.5  -0.866025   
2  unknown         1     -1         0  unknown        0.5  -0.866025   
3  unknown         1     -1         0  unknown        0.5  -0.866025   
4  unknown         1     -1         0  unknown        0.5  -0.866025   

    day_sin   day_cos  
0  0.848644  0.528964  
1  0.848644  0.528964  
2  0.848644  0.528964  
3  0.848644  0.528964  
4  0.848644  0.528964  

Target Variable:

(45195,)


0    0
1    0
2    0
3    0
4    0
Name: y, dtype: int64

------------------------------------

Missing Data Check:

age          0
job          0
marital      0
education    0
default      0
balance      0
housing      0
loan         0
contact      0
campaign     0
pdays        0
previous     0
poutcome     0
y            0
month_sin    0
month_cos    0
day_sin      0
day_cos      0
dtype: int64

------------------------------------

# Splitting the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X_engineered, y_engineered, test_size=0.2, random_state=seed, stratify=y_engineered)

print("--- Train-Test Split ---\n")
print(f"Training set shape: {X_train.shape}, {y_train.shape}")
print(f"Testing set shape: {X_test.shape}, {y_test.shape}")

--- Train-Test Split ---

Training set shape: (36156, 17), (36156,)
Testing set shape: (9039, 17), (9039,)

# --- Model Building and Evaluation ---
# Identifying categorical and numerical features for preprocessing
categorical_features_engineered = X_train.select_dtypes(include=['object']).columns
numerical_features_engineered = X_train.select_dtypes(include=np.number).columns
print("\nCategorical Features (Engineered):\n", categorical_features_engineered)
print("\nNumerical Features (Engineered):\n", numerical_features_engineered)
print("\n------------------------------------\n")

# Selecting features for PowerTransformer
# (Excluding cyclical numerical features)
cyclical_features = ['month_sin', 'month_cos', 'day_sin', 'day_cos']
numerical_features_engineered = [col for col in numerical_features_engineered if col not in cyclical_features]
print("\nNumerical Features for PowerTransformer (Engineered):\n", numerical_features_engineered)

# Creating preprocessor for encoding categorical features and scaling numerical features
preprocessor_engineered = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features_engineered),
        ('num', PowerTransformer(method='yeo-johnson'), numerical_features_engineered),
        ('cyclical', 'passthrough', cyclical_features)
    ],
    remainder='passthrough'
)
print("\nPreprocessor (Engineered):\n", preprocessor_engineered)
print("\n------------------------------\n")

# Defining models to compare
models = {
    "Logistic Regression": LogisticRegression(class_weight='balanced', random_state=seed, max_iter=1000),
    "Random Forest": RandomForestClassifier(class_weight='balanced', random_state=seed),
    "HistGradient Boosting": HistGradientBoostingClassifier(random_state=seed, class_weight='balanced'),
    "XGBoost": XGBClassifier(eval_metric='logloss', random_state=seed,
    scale_pos_weight=(y_train == 0).sum() / (y_train == 1).sum()),
    "LightGBM": LGBMClassifier(random_state=seed, verbose=-1, class_weight='balanced')
}

# Defining Cross-Validation strategy
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)

# Evaluating each model using cross-validation
results = []
roc_plot_data = []

print("\n--- Model Evaluation with Cross-Validation ---\n")
for model_name, model in models.items():
    pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor_engineered),
        ('classifier', model)
    ])

    start_time = time.time()

    cv_results = cross_validate(
        pipeline, X_train, y_train,
        cv=cv,
        scoring=['precision', 'recall', 'f1', 'roc_auc'],
        return_train_score=False)

    end_time = time.time()
    training_time = end_time - start_time

    # Store results
    results.append({
        'Model': model_name,
        'Precision': np.mean(cv_results['test_precision']),
        'Recall': np.mean(cv_results['test_recall']),
        'F1-Score': np.mean(cv_results['test_f1']),
        'ROC AUC': np.mean(cv_results['test_roc_auc']),
        'Training Time (s)': training_time})

    # Fit the model to get ROC curve data
    pipeline.fit(X_train, y_train)
    y_proba = pipeline.predict_proba(X_test)[:, 1]
    y_pred = pipeline.predict(X_test)
    plot_confusion_matrix(y_test, y_pred, model_name)
    roc_plot_data.append({
        'model_name': model_name,
        'y_true': y_test,
        'y_proba': pipeline.predict_proba(X_test)[:, 1]
    })
    print(f"{model_name} evaluated in {training_time:.2f} seconds.")

# Creating a results dataframe
results_df = pd.DataFrame(results)
print("\nCross-Validation Results:\n")
display(results_df.style.format({'Precision': '{:.2f}', 'Recall': '{:.2f}', 'F1-Score': '{:.2f}',
                                 'ROC AUC': '{:.2f}', 'Training Time (s)': '{:.2f}'}))

# Plotting ROC Curves for all models
fig = go.Figure()
colors = px.colors.qualitative.Plotly

fig.add_trace(go.Scatter(
    x=[0, 1], 
    y=[0, 1],
    mode='lines',
    line=dict(color='black', width=2, dash='dash'),
    name='Random Guess'
))

for i, data in enumerate(roc_plot_data):
    y_true = data['y_true']
    y_proba = data['y_proba']
    model_name = data['model_name']
    
    fpr, tpr, _ = roc_curve(y_true, y_proba, pos_label=1)
    roc_auc = auc(fpr, tpr)
    
    fig.add_trace(go.Scatter(
        x=fpr, 
        y=tpr,
        mode='lines',
        line=dict(color=colors[i % len(colors)], width=2),
        name=f"{model_name} (AUC = {roc_auc:.2f})"
    ))

fig.update_layout(
    title='<b>ROC Curve Comparison for All Models</b>',
    xaxis_title='False Positive Rate',
    yaxis_title='True Positive Rate',
    xaxis=dict(range=[0.0, 1.0]),
    yaxis=dict(range=[0.0, 1.05]),
    width=800,
    height=800,
    plot_bgcolor='#f5f7f6',
    paper_bgcolor='#f5f7f6',
    legend=dict(x=0.55, y=0.1),
    xaxis_scaleanchor="y",
    yaxis_scaleratio=1
)

fig.show()

Categorical Features (Engineered):
 Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'poutcome'],
      dtype='object')

Numerical Features (Engineered):
 Index(['age', 'balance', 'campaign', 'pdays', 'previous', 'month_sin',
       'month_cos', 'day_sin', 'day_cos'],
      dtype='object')

------------------------------------


Numerical Features for PowerTransformer (Engineered):
 ['age', 'balance', 'campaign', 'pdays', 'previous']

Preprocessor (Engineered):
 ColumnTransformer(remainder='passthrough',
                  transformers=[('cat', OneHotEncoder(handle_unknown='ignore'),
                                 Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'poutcome'],
      dtype='object')),
                                ('num', PowerTransformer(),
                                 ['age', 'balance', 'campaign', 'pdays',
                                  'previous']),
                                ('cyclical', 'passthrough',
                                 ['month_sin', 'month_cos', 'day_sin',
                                  'day_cos'])])

------------------------------


--- Model Evaluation with Cross-Validation ---

Logistic Regression evaluated in 0.80 seconds.

Random Forest evaluated in 10.08 seconds.

HistGradient Boosting evaluated in 3.43 seconds.

XGBoost evaluated in 1.36 seconds.

LightGBM evaluated in 2.82 seconds.

Cross-Validation Results:

	Model	Precision	Recall	F1-Score	ROC AUC	Training Time (s)
0	Logistic Regression	0.23	0.65	0.34	0.75	0.80
1	Random Forest	0.65	0.20	0.31	0.78	10.08
2	HistGradient Boosting	0.35	0.62	0.45	0.80	3.43
3	XGBoost	0.34	0.54	0.42	0.77	1.36
4	LightGBM	0.35	0.62	0.45	0.80	2.82

• HistGradientBoosting and LightGBM are the top performers, achieving the highest F1-Score (0.45) and ROC AUC (0.80).
  
  
• The XGBoost also achieved a good performance with F1-Score (0.42).
  
  
• The main reason why we're using the F1-Score as the primary metric is because it balances the two competing goals of a typical marketing campaign: Precision (minimizing wasted calls to uninterested clients) and Recall(maximizing the number of potential subscribers we find).

Tuning Models

To improve performance, we will fine-tune the LightGBM, HistGradient Boosting, and XGBoost models with Optuna. The goal is to find the ideal set of hyperparameters to increase performance across these models.

# --- Hyperparameter Tuning with Optuna for LightGBM ---
def objective_lgbm(trial):
    """
    This is the objective function that Optuna will optimize.
    A 'trial' is a single run of the model with a specific set of hyperparameters.
    """
    scale_pos_weight_value = y_train.value_counts()[0] / y_train.value_counts()[1]

    params = {
        'objective': 'binary',
        'metric': 'binary_logloss',
        'boosting_type': trial.suggest_categorical('boosting_type', ['gbdt', 'dart', 'goss']),
        'scale_pos_weight': trial.suggest_float(
            'scale_pos_weight', scale_pos_weight_value * 0.8, scale_pos_weight_value * 1.2
        ),
        'n_estimators': trial.suggest_int('n_estimators', 200, 2000, step=100),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),
        'num_leaves': trial.suggest_int('num_leaves', 20, 3000, step=20),
        'max_depth': trial.suggest_int('max_depth', 3, 12),
        'min_child_samples': trial.suggest_int('min_child_samples', 1, 200),
        'subsample': trial.suggest_float('subsample', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
        'random_state': seed,
        'n_jobs': -1,
        'verbose': -1
    }

    pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor_engineered),
        ('classifier', LGBMClassifier(**params))
    ])

    cv_strategy = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
    score = cross_val_score(
        pipeline, X_train, y_train, cv=cv_strategy, scoring='f1', n_jobs=-1
    ).mean()
    return score

study_lgbm = optuna.create_study(direction='maximize', study_name='LGBM Classifier Optimization')
print("\n--- Starting Hyperparameter Optimization for LightGBM ---\n")
study_lgbm.optimize(objective_lgbm, n_trials=100, show_progress_bar=True)
print("\n--- Hyperparameter Optimization Complete ---\n")
print("Best trial:")
trial = study_lgbm.best_trial
print(f"  Value: {trial.value}")
print("  Params: ")
for key, value in trial.params.items():
    print(f"    {key}: {value}")

[I 2025-09-20 23:41:55,413] A new study created in memory with name: LGBM Classifier Optimization

--- Starting Hyperparameter Optimization for LightGBM ---

  0%|          | 0/100 [00:00<?, ?it/s]

[I 2025-09-20 23:42:49,261] Trial 0 finished with value: 0.3908633995408076 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.389366674104825, 'n_estimators': 1600, 'learning_rate': 0.16225836479058564, 'num_leaves': 1060, 'max_depth': 10, 'min_child_samples': 41, 'subsample': 0.8113756223940406, 'colsample_bytree': 0.7478848326616979, 'reg_alpha': 0.003838891111849317, 'reg_lambda': 1.5810125665564527}. Best is trial 0 with value: 0.3908633995408076.
[I 2025-09-20 23:45:54,032] Trial 1 finished with value: 0.4286612922334876 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.705879947403257, 'n_estimators': 1600, 'learning_rate': 0.20149913898964722, 'num_leaves': 2320, 'max_depth': 10, 'min_child_samples': 190, 'subsample': 0.8437360748227157, 'colsample_bytree': 0.9050138155324632, 'reg_alpha': 1.8136043696243234e-06, 'reg_lambda': 6.077701040045471}. Best is trial 1 with value: 0.4286612922334876.
[I 2025-09-20 23:46:21,364] Trial 2 finished with value: 0.39694451621062454 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 6.665161195549643, 'n_estimators': 1200, 'learning_rate': 0.12746196183243053, 'num_leaves': 980, 'max_depth': 8, 'min_child_samples': 42, 'subsample': 0.8398154972441867, 'colsample_bytree': 0.9871723866599618, 'reg_alpha': 0.0001310848969405817, 'reg_lambda': 0.04094550410182813}. Best is trial 1 with value: 0.4286612922334876.
[I 2025-09-20 23:47:10,425] Trial 3 finished with value: 0.43393348989012326 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.521827575130863, 'n_estimators': 1800, 'learning_rate': 0.1783881765514178, 'num_leaves': 1000, 'max_depth': 3, 'min_child_samples': 142, 'subsample': 0.6287784460082639, 'colsample_bytree': 0.9916574606477723, 'reg_alpha': 1.9067055416534865e-05, 'reg_lambda': 0.20629011991631982}. Best is trial 3 with value: 0.43393348989012326.
[I 2025-09-20 23:48:59,991] Trial 4 finished with value: 0.4298654270769505 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 8.799348949717738, 'n_estimators': 1400, 'learning_rate': 0.1203181961886615, 'num_leaves': 2240, 'max_depth': 8, 'min_child_samples': 146, 'subsample': 0.7647782618446279, 'colsample_bytree': 0.8478975764199488, 'reg_alpha': 6.823310943367778e-06, 'reg_lambda': 0.18875859762438543}. Best is trial 3 with value: 0.43393348989012326.
[I 2025-09-20 23:49:51,789] Trial 5 finished with value: 0.40673596428670766 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 7.630252090008904, 'n_estimators': 1100, 'learning_rate': 0.242684488052315, 'num_leaves': 560, 'max_depth': 6, 'min_child_samples': 164, 'subsample': 0.7625239695402118, 'colsample_bytree': 0.6900971580033622, 'reg_alpha': 0.0002117495742826822, 'reg_lambda': 1.1176028306309447}. Best is trial 3 with value: 0.43393348989012326.
[I 2025-09-20 23:50:44,611] Trial 6 finished with value: 0.4453022078438861 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.5786519446453, 'n_estimators': 800, 'learning_rate': 0.023503397290604407, 'num_leaves': 2660, 'max_depth': 6, 'min_child_samples': 186, 'subsample': 0.6835007008398623, 'colsample_bytree': 0.6217298136902042, 'reg_alpha': 1.8281264604788061, 'reg_lambda': 0.6730192907392493}. Best is trial 6 with value: 0.4453022078438861.
[I 2025-09-20 23:51:03,092] Trial 7 finished with value: 0.38466708413655115 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 8.140511870659173, 'n_estimators': 1900, 'learning_rate': 0.23374698785232456, 'num_leaves': 1360, 'max_depth': 5, 'min_child_samples': 57, 'subsample': 0.7007937687244815, 'colsample_bytree': 0.6874283693800413, 'reg_alpha': 0.1411719153593019, 'reg_lambda': 3.548944462642671}. Best is trial 6 with value: 0.4453022078438861.
[I 2025-09-20 23:52:51,771] Trial 8 finished with value: 0.4317699858930443 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 8.269558131969236, 'n_estimators': 800, 'learning_rate': 0.037169653828466626, 'num_leaves': 2700, 'max_depth': 7, 'min_child_samples': 179, 'subsample': 0.6788081196513723, 'colsample_bytree': 0.9697224338060794, 'reg_alpha': 0.8089289421745635, 'reg_lambda': 2.065719170305385}. Best is trial 6 with value: 0.4453022078438861.
[I 2025-09-20 23:53:32,214] Trial 9 finished with value: 0.38608437908511334 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 8.410151265854863, 'n_estimators': 1700, 'learning_rate': 0.25109226699302856, 'num_leaves': 1420, 'max_depth': 9, 'min_child_samples': 104, 'subsample': 0.6424058878407642, 'colsample_bytree': 0.7214510666534417, 'reg_alpha': 0.0012616281367873194, 'reg_lambda': 2.5544229499514392e-05}. Best is trial 6 with value: 0.4453022078438861.
[I 2025-09-20 23:55:17,319] Trial 10 finished with value: 0.4778835259113253 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.972298086453031, 'n_estimators': 200, 'learning_rate': 0.014834862299224885, 'num_leaves': 2860, 'max_depth': 12, 'min_child_samples': 102, 'subsample': 0.965526542432651, 'colsample_bytree': 0.6121688220054424, 'reg_alpha': 2.476588268959195e-08, 'reg_lambda': 1.2795939632842592e-08}. Best is trial 10 with value: 0.4778835259113253.
[I 2025-09-20 23:55:39,484] Trial 11 finished with value: 0.4817346018383605 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.009971471773609, 'n_estimators': 200, 'learning_rate': 0.011933663386885873, 'num_leaves': 2980, 'max_depth': 12, 'min_child_samples': 99, 'subsample': 0.976117178865086, 'colsample_bytree': 0.6045884605959665, 'reg_alpha': 1.0023526200624856e-08, 'reg_lambda': 1.246535480598537e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-20 23:56:34,287] Trial 12 finished with value: 0.46752150640256096 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.828369275367342, 'n_estimators': 200, 'learning_rate': 0.07568391764374974, 'num_leaves': 3000, 'max_depth': 12, 'min_child_samples': 92, 'subsample': 0.9998903862778655, 'colsample_bytree': 0.6006581051557768, 'reg_alpha': 1.4898564586858517e-08, 'reg_lambda': 1.1297710718903886e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:00:47,247] Trial 13 finished with value: 0.44470955845349 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.045286956392519, 'n_estimators': 200, 'learning_rate': 0.07065371616376508, 'num_leaves': 2080, 'max_depth': 12, 'min_child_samples': 3, 'subsample': 0.9907728125779226, 'colsample_bytree': 0.6469566845043514, 'reg_alpha': 1.1951033202741096e-08, 'reg_lambda': 1.7807394334467362e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:02:47,225] Trial 14 finished with value: 0.4684072946959718 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.068707903020772, 'n_estimators': 500, 'learning_rate': 0.011457697803743849, 'num_leaves': 20, 'max_depth': 11, 'min_child_samples': 101, 'subsample': 0.9163326667925002, 'colsample_bytree': 0.8028610920160697, 'reg_alpha': 3.583007904673016e-07, 'reg_lambda': 4.549927623073226e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:07:05,218] Trial 15 finished with value: 0.45925448441004696 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.099963267362335, 'n_estimators': 500, 'learning_rate': 0.06981801460423794, 'num_leaves': 1840, 'max_depth': 12, 'min_child_samples': 126, 'subsample': 0.9256712036598802, 'colsample_bytree': 0.6583528388511253, 'reg_alpha': 1.0677327133252747e-07, 'reg_lambda': 9.505639423503221e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:22:52,343] Trial 16 finished with value: 0.45374125162991363 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.2547961685369025, 'n_estimators': 500, 'learning_rate': 0.10281548089470556, 'num_leaves': 3000, 'max_depth': 10, 'min_child_samples': 77, 'subsample': 0.9334231545299166, 'colsample_bytree': 0.7672602931261974, 'reg_alpha': 1.9924951992232286e-07, 'reg_lambda': 0.002279076264323218}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:24:18,592] Trial 17 finished with value: 0.43532144442571746 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.439594004538272, 'n_estimators': 800, 'learning_rate': 0.29088209798310016, 'num_leaves': 2680, 'max_depth': 11, 'min_child_samples': 122, 'subsample': 0.88618841754375, 'colsample_bytree': 0.6025973816185439, 'reg_alpha': 1.0613640685000985e-08, 'reg_lambda': 3.962652977046484e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:24:29,927] Trial 18 finished with value: 0.4467864319652891 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.78620893707345, 'n_estimators': 300, 'learning_rate': 0.04711656370576438, 'num_leaves': 1820, 'max_depth': 11, 'min_child_samples': 69, 'subsample': 0.9606447747677499, 'colsample_bytree': 0.8270323754071219, 'reg_alpha': 0.023862773950047985, 'reg_lambda': 4.712149945220017e-05}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:24:55,254] Trial 19 finished with value: 0.4248273643145182 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 7.284640835235716, 'n_estimators': 400, 'learning_rate': 0.09655512372501798, 'num_leaves': 2420, 'max_depth': 9, 'min_child_samples': 8, 'subsample': 0.8749003750988912, 'colsample_bytree': 0.6853135003837205, 'reg_alpha': 7.296837507554547e-07, 'reg_lambda': 5.447221461348745e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:25:12,218] Trial 20 finished with value: 0.41495279463632 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.845993774164179, 'n_estimators': 700, 'learning_rate': 0.047491351926018416, 'num_leaves': 1920, 'max_depth': 3, 'min_child_samples': 87, 'subsample': 0.9667273854898822, 'colsample_bytree': 0.882522731219487, 'reg_alpha': 1.8380258803138625e-05, 'reg_lambda': 3.689482806919035e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:25:27,545] Trial 21 finished with value: 0.4653099612370951 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.123798104784656, 'n_estimators': 600, 'learning_rate': 0.011329656644576747, 'num_leaves': 320, 'max_depth': 11, 'min_child_samples': 111, 'subsample': 0.92134987198528, 'colsample_bytree': 0.7846355363081997, 'reg_alpha': 1.5989780358245706e-07, 'reg_lambda': 1.4836291669793355e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:25:34,175] Trial 22 finished with value: 0.4770554128607607 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.195816036779988, 'n_estimators': 200, 'learning_rate': 0.012080437887829098, 'num_leaves': 180, 'max_depth': 12, 'min_child_samples': 136, 'subsample': 0.9068511037029782, 'colsample_bytree': 0.7321599953575915, 'reg_alpha': 5.470863232414501e-08, 'reg_lambda': 2.116858578874771e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:25:39,213] Trial 23 finished with value: 0.45606336353500404 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.546633210310291, 'n_estimators': 200, 'learning_rate': 0.04829072016343314, 'num_leaves': 700, 'max_depth': 12, 'min_child_samples': 142, 'subsample': 0.9610598752464595, 'colsample_bytree': 0.6488622626467221, 'reg_alpha': 6.206935471798572e-08, 'reg_lambda': 4.736283961372098e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:25:50,558] Trial 24 finished with value: 0.45766159005305573 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.300123635357001, 'n_estimators': 400, 'learning_rate': 0.03240720625706157, 'num_leaves': 1600, 'max_depth': 12, 'min_child_samples': 120, 'subsample': 0.8911013209849813, 'colsample_bytree': 0.7405892799487963, 'reg_alpha': 2.0441476803836985e-06, 'reg_lambda': 6.437010259460502e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:25:57,064] Trial 25 finished with value: 0.44807736259726544 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.926335051136279, 'n_estimators': 300, 'learning_rate': 0.06823500897468492, 'num_leaves': 2800, 'max_depth': 10, 'min_child_samples': 153, 'subsample': 0.9653540008680931, 'colsample_bytree': 0.7141295155016151, 'reg_alpha': 4.227836948692645e-08, 'reg_lambda': 0.0007730704027352062}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:36:24,800] Trial 26 finished with value: 0.45473011554707926 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.632673111691114, 'n_estimators': 1000, 'learning_rate': 0.09227972140064203, 'num_leaves': 2460, 'max_depth': 11, 'min_child_samples': 165, 'subsample': 0.8548588161256638, 'colsample_bytree': 0.6316469210379163, 'reg_alpha': 4.51797074914143e-08, 'reg_lambda': 2.356564238963638e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:36:58,005] Trial 27 finished with value: 0.46026424015022577 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.360825376816071, 'n_estimators': 400, 'learning_rate': 0.015641039274823885, 'num_leaves': 280, 'max_depth': 9, 'min_child_samples': 130, 'subsample': 0.9445842324236793, 'colsample_bytree': 0.664550862342703, 'reg_alpha': 1.1462693370527094e-06, 'reg_lambda': 3.5822583004677016e-05}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:37:20,682] Trial 28 finished with value: 0.41665631696546984 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 7.972817774731734, 'n_estimators': 600, 'learning_rate': 0.12297278187872768, 'num_leaves': 2140, 'max_depth': 12, 'min_child_samples': 83, 'subsample': 0.8989925605827613, 'colsample_bytree': 0.6233857521819238, 'reg_alpha': 5.4475802551509965e-06, 'reg_lambda': 5.303886961474329e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:37:26,881] Trial 29 finished with value: 0.43585195783168523 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.256284153802947, 'n_estimators': 200, 'learning_rate': 0.15099605060221177, 'num_leaves': 2500, 'max_depth': 10, 'min_child_samples': 61, 'subsample': 0.7941616718980921, 'colsample_bytree': 0.7491926344017726, 'reg_alpha': 0.004116552420132306, 'reg_lambda': 1.0365650105920966e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:37:57,546] Trial 30 finished with value: 0.4264840295775584 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 7.106099747536738, 'n_estimators': 900, 'learning_rate': 0.05666451517744866, 'num_leaves': 740, 'max_depth': 11, 'min_child_samples': 43, 'subsample': 0.9898914020470557, 'colsample_bytree': 0.7129646985270401, 'reg_alpha': 4.678153577506407e-05, 'reg_lambda': 1.213060232725975e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:38:06,863] Trial 31 finished with value: 0.47441206329737484 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.056294976515667, 'n_estimators': 300, 'learning_rate': 0.011191798691379998, 'num_leaves': 140, 'max_depth': 11, 'min_child_samples': 100, 'subsample': 0.9001839421673419, 'colsample_bytree': 0.8267006127724723, 'reg_alpha': 3.0987955145928755e-07, 'reg_lambda': 2.0753124923272753e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:38:15,208] Trial 32 finished with value: 0.460439129102484 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.396146537442363, 'n_estimators': 300, 'learning_rate': 0.02216120315353129, 'num_leaves': 320, 'max_depth': 12, 'min_child_samples': 111, 'subsample': 0.815294273119321, 'colsample_bytree': 0.8402691951490844, 'reg_alpha': 3.656997791563546e-08, 'reg_lambda': 3.184997766782979e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:38:21,945] Trial 33 finished with value: 0.4624081926977698 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.117366742884291, 'n_estimators': 300, 'learning_rate': 0.03417665639491055, 'num_leaves': 80, 'max_depth': 10, 'min_child_samples': 98, 'subsample': 0.8642980953404628, 'colsample_bytree': 0.9264570509405654, 'reg_alpha': 3.903038380801406e-07, 'reg_lambda': 3.841154135113382e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:38:47,410] Trial 34 finished with value: 0.44410507992354364 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.590742189589944, 'n_estimators': 1400, 'learning_rate': 0.03370909986844587, 'num_leaves': 1180, 'max_depth': 11, 'min_child_samples': 200, 'subsample': 0.9025326459100955, 'colsample_bytree': 0.8772563316672171, 'reg_alpha': 2.2340850066298354e-06, 'reg_lambda': 1.1735887923904024e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:40:05,799] Trial 35 finished with value: 0.450967016512986 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.7757862007533785, 'n_estimators': 600, 'learning_rate': 0.1899546071250648, 'num_leaves': 520, 'max_depth': 12, 'min_child_samples': 136, 'subsample': 0.8275831894027752, 'colsample_bytree': 0.810451625976269, 'reg_alpha': 3.3801185672416624e-08, 'reg_lambda': 9.247271824316922e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:40:14,791] Trial 36 finished with value: 0.4105060440961738 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 9.00306606703588, 'n_estimators': 400, 'learning_rate': 0.15637064601468478, 'num_leaves': 840, 'max_depth': 11, 'min_child_samples': 111, 'subsample': 0.9387937913182812, 'colsample_bytree': 0.7731840936943678, 'reg_alpha': 1.9224690223265632e-07, 'reg_lambda': 0.009516650812558916}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:43:38,313] Trial 37 finished with value: 0.45089032938362356 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 7.460636156644982, 'n_estimators': 1400, 'learning_rate': 0.010396972669469422, 'num_leaves': 2860, 'max_depth': 8, 'min_child_samples': 75, 'subsample': 0.7910442214156487, 'colsample_bytree': 0.8728353387755988, 'reg_alpha': 1.2899563462778741e-08, 'reg_lambda': 1.552625402054831e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:43:48,146] Trial 38 finished with value: 0.468982142649367 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.228434129890765, 'n_estimators': 200, 'learning_rate': 0.05561011552257817, 'num_leaves': 160, 'max_depth': 9, 'min_child_samples': 159, 'subsample': 0.9766134487048012, 'colsample_bytree': 0.9229857229651549, 'reg_alpha': 6.6943731818620515e-06, 'reg_lambda': 1.9451031187213356e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:46:36,531] Trial 39 finished with value: 0.46251423369431743 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.427661732115101, 'n_estimators': 1200, 'learning_rate': 0.08255171662011965, 'num_leaves': 520, 'max_depth': 10, 'min_child_samples': 28, 'subsample': 0.9506465390869945, 'colsample_bytree': 0.6781305876795246, 'reg_alpha': 8.196761616861282, 'reg_lambda': 0.00016674805091896177}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:46:39,332] Trial 40 finished with value: 0.428596044340569 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 7.725367840039576, 'n_estimators': 300, 'learning_rate': 0.1367984667830447, 'num_leaves': 1180, 'max_depth': 4, 'min_child_samples': 92, 'subsample': 0.8404829664910046, 'colsample_bytree': 0.6253950004052905, 'reg_alpha': 6.757723829026038e-07, 'reg_lambda': 1.0292118145562308e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:46:47,392] Trial 41 finished with value: 0.4733539208394714 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.237958977775783, 'n_estimators': 200, 'learning_rate': 0.05537425044973544, 'num_leaves': 180, 'max_depth': 7, 'min_child_samples': 157, 'subsample': 0.9796774875736466, 'colsample_bytree': 0.9359541127861792, 'reg_alpha': 3.4117485458838646e-06, 'reg_lambda': 2.658124087801198e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:46:56,400] Trial 42 finished with value: 0.4716636743025645 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.04592186219003, 'n_estimators': 200, 'learning_rate': 0.027444748507140376, 'num_leaves': 200, 'max_depth': 7, 'min_child_samples': 170, 'subsample': 0.9834530736091378, 'colsample_bytree': 0.9733181225880856, 'reg_alpha': 1.005144363269842e-07, 'reg_lambda': 3.0977342837259176e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:47:16,369] Trial 43 finished with value: 0.4736701828121902 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.265992210794432, 'n_estimators': 400, 'learning_rate': 0.03860243741035613, 'num_leaves': 380, 'max_depth': 6, 'min_child_samples': 153, 'subsample': 0.9111376952696468, 'colsample_bytree': 0.9381563195214856, 'reg_alpha': 7.355573948055053e-05, 'reg_lambda': 1.4794081002980064e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:47:43,538] Trial 44 finished with value: 0.4637062067259654 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.946403940329435, 'n_estimators': 500, 'learning_rate': 0.023949644671860386, 'num_leaves': 420, 'max_depth': 6, 'min_child_samples': 148, 'subsample': 0.912680873936667, 'colsample_bytree': 0.9475198476504535, 'reg_alpha': 0.0005909002361053525, 'reg_lambda': 2.677215341978672e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:47:58,560] Trial 45 finished with value: 0.4629680288117582 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.525211399879199, 'n_estimators': 400, 'learning_rate': 0.041069459798762524, 'num_leaves': 660, 'max_depth': 5, 'min_child_samples': 135, 'subsample': 0.7387895327445589, 'colsample_bytree': 0.6061077933248596, 'reg_alpha': 1.4363404811954932e-05, 'reg_lambda': 9.102251126890228e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:48:07,739] Trial 46 finished with value: 0.4589942251893673 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.728819311236716, 'n_estimators': 300, 'learning_rate': 0.019492544311509506, 'num_leaves': 840, 'max_depth': 5, 'min_child_samples': 115, 'subsample': 0.8757165767036421, 'colsample_bytree': 0.8988281994451577, 'reg_alpha': 0.00016024146092077728, 'reg_lambda': 1.1414079673390731e-05}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:48:13,523] Trial 47 finished with value: 0.4188996127876603 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 8.562113221486978, 'n_estimators': 700, 'learning_rate': 0.10920019301828204, 'num_leaves': 20, 'max_depth': 6, 'min_child_samples': 179, 'subsample': 0.9372266678799859, 'colsample_bytree': 0.8532307237897425, 'reg_alpha': 6.625433526883656e-05, 'reg_lambda': 5.941868185495818e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:56:17,459] Trial 48 finished with value: 0.44937887451576924 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.346735849416835, 'n_estimators': 2000, 'learning_rate': 0.06193795610922391, 'num_leaves': 2600, 'max_depth': 12, 'min_child_samples': 101, 'subsample': 0.908972084436214, 'colsample_bytree': 0.6982471669270762, 'reg_alpha': 2.070058287640824e-08, 'reg_lambda': 1.0837304457984667e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:56:32,944] Trial 49 finished with value: 0.45343177652716243 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.9247145117013, 'n_estimators': 500, 'learning_rate': 0.2234261706978352, 'num_leaves': 960, 'max_depth': 4, 'min_child_samples': 133, 'subsample': 0.9970477763975472, 'colsample_bytree': 0.7449519689407886, 'reg_alpha': 3.8270146998037117e-07, 'reg_lambda': 2.2720604410300284e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:57:01,995] Trial 50 finished with value: 0.4437496224721865 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 7.183658021830501, 'n_estimators': 700, 'learning_rate': 0.03931985457922234, 'num_leaves': 1620, 'max_depth': 12, 'min_child_samples': 57, 'subsample': 0.9236348290054132, 'colsample_bytree': 0.6427260182265696, 'reg_alpha': 0.0810121106292597, 'reg_lambda': 1.5266637986983478e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:57:12,038] Trial 51 finished with value: 0.4619642170889479 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.198977877012388, 'n_estimators': 200, 'learning_rate': 0.010190083677980922, 'num_leaves': 180, 'max_depth': 8, 'min_child_samples': 157, 'subsample': 0.9764931190837668, 'colsample_bytree': 0.9317956144708789, 'reg_alpha': 3.0760338543174146e-06, 'reg_lambda': 2.485918517004387e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:57:24,258] Trial 52 finished with value: 0.4741216467099349 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.180961519273127, 'n_estimators': 300, 'learning_rate': 0.08340001396987726, 'num_leaves': 480, 'max_depth': 7, 'min_child_samples': 176, 'subsample': 0.949755058691661, 'colsample_bytree': 0.9935833633468747, 'reg_alpha': 8.925896296660971e-08, 'reg_lambda': 3.841764020839698e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:57:43,691] Trial 53 finished with value: 0.47385686391470394 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.037007832879555, 'n_estimators': 400, 'learning_rate': 0.08514302981921812, 'num_leaves': 360, 'max_depth': 7, 'min_child_samples': 178, 'subsample': 0.9473714120276282, 'colsample_bytree': 0.9955448105757545, 'reg_alpha': 8.897727657011692e-08, 'reg_lambda': 2.731813299061791e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:57:56,724] Trial 54 finished with value: 0.47446840363455384 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.042617237521299, 'n_estimators': 300, 'learning_rate': 0.11252521125596274, 'num_leaves': 2860, 'max_depth': 7, 'min_child_samples': 197, 'subsample': 0.953380887207415, 'colsample_bytree': 0.9927635464921869, 'reg_alpha': 9.81541477542295e-08, 'reg_lambda': 6.163303371261429e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 00:58:12,418] Trial 55 finished with value: 0.4678366068292917 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.483977299033431, 'n_estimators': 300, 'learning_rate': 0.11373771775184502, 'num_leaves': 2840, 'max_depth': 8, 'min_child_samples': 193, 'subsample': 0.9569181466334935, 'colsample_bytree': 0.965815982900934, 'reg_alpha': 2.5205814478267855e-07, 'reg_lambda': 5.693536405760272e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:00:15,319] Trial 56 finished with value: 0.4455394640289175 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.14539355476443, 'n_estimators': 1600, 'learning_rate': 0.17097279485237063, 'num_leaves': 2980, 'max_depth': 7, 'min_child_samples': 173, 'subsample': 0.9316629857916626, 'colsample_bytree': 0.977767116399017, 'reg_alpha': 3.158078244555708e-08, 'reg_lambda': 1.1915345738657087e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:00:20,972] Trial 57 finished with value: 0.45085333311151093 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.355211697351659, 'n_estimators': 300, 'learning_rate': 0.07815969702050217, 'num_leaves': 2280, 'max_depth': 8, 'min_child_samples': 186, 'subsample': 0.882856378189548, 'colsample_bytree': 0.9993718986458946, 'reg_alpha': 1.114024074190562e-08, 'reg_lambda': 0.21645590838592985}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:00:36,564] Trial 58 finished with value: 0.4266511605254412 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 6.681645312616285, 'n_estimators': 600, 'learning_rate': 0.1338860767986109, 'num_leaves': 2580, 'max_depth': 12, 'min_child_samples': 192, 'subsample': 0.6079499769088907, 'colsample_bytree': 0.9596753370551577, 'reg_alpha': 7.93610651818759e-08, 'reg_lambda': 5.327615883669068e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:01:40,312] Trial 59 finished with value: 0.4493639266950404 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.447762693133359, 'n_estimators': 500, 'learning_rate': 0.02417480653787421, 'num_leaves': 2760, 'max_depth': 11, 'min_child_samples': 91, 'subsample': 0.9699987574953146, 'colsample_bytree': 0.9068484930056738, 'reg_alpha': 8.147324753488401e-07, 'reg_lambda': 6.301317331088513e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:01:57,826] Trial 60 finished with value: 0.4724197099339378 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.168278818464557, 'n_estimators': 200, 'learning_rate': 0.09790604899061758, 'num_leaves': 2960, 'max_depth': 11, 'min_child_samples': 106, 'subsample': 0.9524286302599209, 'colsample_bytree': 0.7279882513758453, 'reg_alpha': 1.2945624018147537e-07, 'reg_lambda': 1.723397800729304e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:02:19,763] Trial 61 finished with value: 0.474183554346005 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.042427125285383, 'n_estimators': 400, 'learning_rate': 0.08680331996060188, 'num_leaves': 2360, 'max_depth': 7, 'min_child_samples': 181, 'subsample': 0.9464200773908191, 'colsample_bytree': 0.9889204748058964, 'reg_alpha': 2.2895730518113085e-08, 'reg_lambda': 2.024753757261346e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:02:33,191] Trial 62 finished with value: 0.4727692081302942 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.056971141217131, 'n_estimators': 300, 'learning_rate': 0.14475920049574387, 'num_leaves': 2380, 'max_depth': 7, 'min_child_samples': 184, 'subsample': 0.9263930176929436, 'colsample_bytree': 0.9831793711485632, 'reg_alpha': 3.7786619370028593e-08, 'reg_lambda': 7.576455971786453}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:02:51,560] Trial 63 finished with value: 0.46804340999994726 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.359236436986924, 'n_estimators': 400, 'learning_rate': 0.06531820592092002, 'num_leaves': 2900, 'max_depth': 6, 'min_child_samples': 200, 'subsample': 0.8933370939058313, 'colsample_bytree': 0.6100389985468057, 'reg_alpha': 2.2553469882461597e-08, 'reg_lambda': 3.3391799964688806e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:03:03,246] Trial 64 finished with value: 0.46858779994080973 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.489062287481005, 'n_estimators': 200, 'learning_rate': 0.046753375691560184, 'num_leaves': 2120, 'max_depth': 8, 'min_child_samples': 122, 'subsample': 0.9615882530666561, 'colsample_bytree': 0.9563226568986243, 'reg_alpha': 7.587026609171273e-08, 'reg_lambda': 1.4463399283964375e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:03:11,889] Trial 65 finished with value: 0.44660032461855315 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.152863053042094, 'n_estimators': 500, 'learning_rate': 0.08809636940670158, 'num_leaves': 2760, 'max_depth': 7, 'min_child_samples': 167, 'subsample': 0.9945321987050451, 'colsample_bytree': 0.6675834703854998, 'reg_alpha': 2.341918890146582e-08, 'reg_lambda': 8.797074156124518e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:03:49,949] Trial 66 finished with value: 0.4635917162441442 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.614008748107877, 'n_estimators': 300, 'learning_rate': 0.10815823483311694, 'num_leaves': 2660, 'max_depth': 12, 'min_child_samples': 77, 'subsample': 0.8535764377274971, 'colsample_bytree': 0.9845914224750423, 'reg_alpha': 2.58904630649798e-07, 'reg_lambda': 2.428833191324872e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:03:53,645] Trial 67 finished with value: 0.4663713292154658 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.288890299606666, 'n_estimators': 200, 'learning_rate': 0.02913611305354372, 'num_leaves': 2900, 'max_depth': 6, 'min_child_samples': 97, 'subsample': 0.9481716807994732, 'colsample_bytree': 0.7950326822797139, 'reg_alpha': 1.0429040574184347e-08, 'reg_lambda': 6.083199644648167e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:04:51,035] Trial 68 finished with value: 0.4492388223999094 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 8.136554573020987, 'n_estimators': 400, 'learning_rate': 0.11708839683827621, 'num_leaves': 2540, 'max_depth': 12, 'min_child_samples': 84, 'subsample': 0.9350119240104903, 'colsample_bytree': 0.7605549693918998, 'reg_alpha': 5.7808376379144935e-08, 'reg_lambda': 7.526935409359851e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:05:00,551] Trial 69 finished with value: 0.4156162059711571 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 6.138967672830816, 'n_estimators': 300, 'learning_rate': 0.26536065680920284, 'num_leaves': 2700, 'max_depth': 11, 'min_child_samples': 141, 'subsample': 0.9656301956457258, 'colsample_bytree': 0.8249879523389476, 'reg_alpha': 4.947243648364881e-07, 'reg_lambda': 3.903721835363571e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:05:09,390] Trial 70 finished with value: 0.4534158361162096 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.847598967655184, 'n_estimators': 500, 'learning_rate': 0.01808508871737432, 'num_leaves': 1980, 'max_depth': 7, 'min_child_samples': 127, 'subsample': 0.920657303841117, 'colsample_bytree': 0.6335585092023528, 'reg_alpha': 1.4958263736736013e-07, 'reg_lambda': 5.677620846925829e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:05:28,996] Trial 71 finished with value: 0.47448335926150353 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.046548500517057, 'n_estimators': 400, 'learning_rate': 0.08692599733323239, 'num_leaves': 460, 'max_depth': 7, 'min_child_samples': 184, 'subsample': 0.9433563227029108, 'colsample_bytree': 0.9951957715758255, 'reg_alpha': 1.2319205856106805e-06, 'reg_lambda': 2.6708811013371153e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:07:29,946] Trial 72 finished with value: 0.46140125614907 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.222167612693716, 'n_estimators': 1500, 'learning_rate': 0.07352031892456949, 'num_leaves': 480, 'max_depth': 7, 'min_child_samples': 174, 'subsample': 0.9853545371855855, 'colsample_bytree': 0.9887043705286785, 'reg_alpha': 5.54337963020383e-08, 'reg_lambda': 1.929133588143723e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:07:39,093] Trial 73 finished with value: 0.47268248092279536 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.049499601042163, 'n_estimators': 200, 'learning_rate': 0.09710969270707277, 'num_leaves': 640, 'max_depth': 8, 'min_child_samples': 192, 'subsample': 0.9004707444094714, 'colsample_bytree': 0.9573048075112027, 'reg_alpha': 1.2216672892215397e-06, 'reg_lambda': 1.91903924247587e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:07:57,380] Trial 74 finished with value: 0.4685190981801305 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.440112443053664, 'n_estimators': 300, 'learning_rate': 0.049781986151414194, 'num_leaves': 120, 'max_depth': 9, 'min_child_samples': 181, 'subsample': 0.9398022245495752, 'colsample_bytree': 0.970811908472532, 'reg_alpha': 1.9932121686467176e-08, 'reg_lambda': 4.410503295370936e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:09:06,263] Trial 75 finished with value: 0.465622856117483 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.299211195863366, 'n_estimators': 600, 'learning_rate': 0.06105419998270087, 'num_leaves': 240, 'max_depth': 12, 'min_child_samples': 187, 'subsample': 0.9701901613875502, 'colsample_bytree': 0.9477946722908764, 'reg_alpha': 1.901360328694028e-07, 'reg_lambda': 0.00013183801361928873}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:09:23,123] Trial 76 finished with value: 0.4440393776347185 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.684448163719725, 'n_estimators': 400, 'learning_rate': 0.12893494046309556, 'num_leaves': 580, 'max_depth': 6, 'min_child_samples': 196, 'subsample': 0.953674896587267, 'colsample_bytree': 0.6138542586342486, 'reg_alpha': 1.8330603342645983e-08, 'reg_lambda': 1.6626352971548245e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:09:26,397] Trial 77 finished with value: 0.4551430334506307 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.125177429785592, 'n_estimators': 200, 'learning_rate': 0.10429426108081462, 'num_leaves': 1780, 'max_depth': 7, 'min_child_samples': 164, 'subsample': 0.9998201071380485, 'colsample_bytree': 0.9983431356744339, 'reg_alpha': 0.00405595135067934, 'reg_lambda': 8.92241023471733e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:10:00,065] Trial 78 finished with value: 0.45882437166355255 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.052388267035183, 'n_estimators': 300, 'learning_rate': 0.016062190342765865, 'num_leaves': 2220, 'max_depth': 12, 'min_child_samples': 117, 'subsample': 0.917043962385063, 'colsample_bytree': 0.9143935255144122, 'reg_alpha': 4.3311627394741826e-08, 'reg_lambda': 8.377993639432086e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:10:27,943] Trial 79 finished with value: 0.470086274250315 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.3401265546641845, 'n_estimators': 400, 'learning_rate': 0.07836321716773441, 'num_leaves': 1360, 'max_depth': 8, 'min_child_samples': 105, 'subsample': 0.9822725790687515, 'colsample_bytree': 0.864727473571256, 'reg_alpha': 1.1529213452914418e-07, 'reg_lambda': 4.3881597685767525e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:10:56,762] Trial 80 finished with value: 0.44934399690151433 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.24001677142815, 'n_estimators': 1100, 'learning_rate': 0.029569962097230756, 'num_leaves': 60, 'max_depth': 11, 'min_child_samples': 65, 'subsample': 0.656792511060664, 'colsample_bytree': 0.6517738036194473, 'reg_alpha': 1.5076302733051974e-06, 'reg_lambda': 0.018050672413358834}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:11:16,938] Trial 81 finished with value: 0.47009563810651284 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.061965245869842, 'n_estimators': 400, 'learning_rate': 0.0866314946352638, 'num_leaves': 300, 'max_depth': 7, 'min_child_samples': 183, 'subsample': 0.9518546132958251, 'colsample_bytree': 0.9919096743387064, 'reg_alpha': 8.518387027613524e-08, 'reg_lambda': 3.535444125130817e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:11:24,091] Trial 82 finished with value: 0.4740826849815285 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.039196969894007, 'n_estimators': 200, 'learning_rate': 0.08899310325356544, 'num_leaves': 460, 'max_depth': 7, 'min_child_samples': 175, 'subsample': 0.9456564375869368, 'colsample_bytree': 0.9829557171967451, 'reg_alpha': 3.349186286508964e-07, 'reg_lambda': 2.175251813078956e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:11:30,280] Trial 83 finished with value: 0.47260977636916623 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.190097928517408, 'n_estimators': 200, 'learning_rate': 0.09172996717903584, 'num_leaves': 780, 'max_depth': 6, 'min_child_samples': 172, 'subsample': 0.9276151288227044, 'colsample_bytree': 0.975669144796685, 'reg_alpha': 4.7312306198992235e-07, 'reg_lambda': 1.6290405325811096e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:11:43,113] Trial 84 finished with value: 0.469982110182473 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.431507270779134, 'n_estimators': 300, 'learning_rate': 0.07180657812216222, 'num_leaves': 420, 'max_depth': 7, 'min_child_samples': 147, 'subsample': 0.9418693058157543, 'colsample_bytree': 0.943991278917726, 'reg_alpha': 2.9925871640906005e-07, 'reg_lambda': 1.191171110642891e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:11:51,805] Trial 85 finished with value: 0.4698688418136775 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.115726991639769, 'n_estimators': 200, 'learning_rate': 0.2002921724270186, 'num_leaves': 580, 'max_depth': 8, 'min_child_samples': 176, 'subsample': 0.9734848877935375, 'colsample_bytree': 0.6959935650621547, 'reg_alpha': 7.638094888010757e-07, 'reg_lambda': 2.1431845109321007e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:12:01,782] Trial 86 finished with value: 0.4666429732463434 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.537689620257079, 'n_estimators': 300, 'learning_rate': 0.12281131937466547, 'num_leaves': 260, 'max_depth': 6, 'min_child_samples': 197, 'subsample': 0.9069317497585995, 'colsample_bytree': 0.965086464958093, 'reg_alpha': 2.9339450307528103e-08, 'reg_lambda': 2.974941895975945e-06}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:12:09,635] Trial 87 finished with value: 0.47393465972761994 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.288348448487402, 'n_estimators': 200, 'learning_rate': 0.04065151101724114, 'num_leaves': 460, 'max_depth': 7, 'min_child_samples': 161, 'subsample': 0.732942872205296, 'colsample_bytree': 0.9763599930176224, 'reg_alpha': 1.644181718790472e-07, 'reg_lambda': 2.6225633513980418e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:12:20,102] Trial 88 finished with value: 0.4410301743515757 and parameters: {'boosting_type': 'gbdt', 'scale_pos_weight': 6.110077666976065, 'n_estimators': 500, 'learning_rate': 0.10236913436057524, 'num_leaves': 3000, 'max_depth': 10, 'min_child_samples': 188, 'subsample': 0.959810949237052, 'colsample_bytree': 0.987453449201529, 'reg_alpha': 5.344471941021914e-08, 'reg_lambda': 4.453663618773916e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:12:28,213] Trial 89 finished with value: 0.4443368057856475 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 7.234840221399131, 'n_estimators': 300, 'learning_rate': 0.034170527049218316, 'num_leaves': 2360, 'max_depth': 12, 'min_child_samples': 168, 'subsample': 0.8705525748497492, 'colsample_bytree': 0.8919375084768036, 'reg_alpha': 1.0180060091072569e-08, 'reg_lambda': 8.112327728817786e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:12:38,916] Trial 90 finished with value: 0.4728433080904918 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.187728980191108, 'n_estimators': 200, 'learning_rate': 0.053865639427849575, 'num_leaves': 120, 'max_depth': 7, 'min_child_samples': 96, 'subsample': 0.930403560857533, 'colsample_bytree': 0.6360655934968602, 'reg_alpha': 1.910317758770375e-08, 'reg_lambda': 1.1720611305684623e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:12:47,573] Trial 91 finished with value: 0.47590171925162184 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.286321142214762, 'n_estimators': 200, 'learning_rate': 0.042408126846208455, 'num_leaves': 460, 'max_depth': 7, 'min_child_samples': 161, 'subsample': 0.6985475004521293, 'colsample_bytree': 0.9796063770353234, 'reg_alpha': 2.0876078881055852e-07, 'reg_lambda': 2.337631027776297e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:13:08,764] Trial 92 finished with value: 0.47344642867124664 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.040608653150891, 'n_estimators': 400, 'learning_rate': 0.017722570990426612, 'num_leaves': 360, 'max_depth': 7, 'min_child_samples': 179, 'subsample': 0.7446220049942863, 'colsample_bytree': 0.9826525506193909, 'reg_alpha': 2.782912579638536e-07, 'reg_lambda': 1.260335660236746e-07}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:13:25,157] Trial 93 finished with value: 0.463324528938021 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.400262446246742, 'n_estimators': 300, 'learning_rate': 0.01369914059539327, 'num_leaves': 940, 'max_depth': 8, 'min_child_samples': 162, 'subsample': 0.7076538471183411, 'colsample_bytree': 0.9989925501091312, 'reg_alpha': 1.2233582332146498e-07, 'reg_lambda': 7.517639926464168e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:13:30,531] Trial 94 finished with value: 0.45304999741428437 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.241368244407031, 'n_estimators': 200, 'learning_rate': 0.023131465438049947, 'num_leaves': 1120, 'max_depth': 5, 'min_child_samples': 151, 'subsample': 0.7805485088677624, 'colsample_bytree': 0.9547422269606871, 'reg_alpha': 3.227473310097345e-08, 'reg_lambda': 2.417369931487073e-08}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:14:44,075] Trial 95 finished with value: 0.47635639554635556 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.099929215801902, 'n_estimators': 900, 'learning_rate': 0.010478592340287255, 'num_leaves': 2820, 'max_depth': 7, 'min_child_samples': 89, 'subsample': 0.8243429232603072, 'colsample_bytree': 0.969716898706688, 'reg_alpha': 1.1513366193094707e-05, 'reg_lambda': 0.000714085431512782}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:16:27,736] Trial 96 finished with value: 0.4674858988700429 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.309862124441212, 'n_estimators': 1200, 'learning_rate': 0.043662520540259236, 'num_leaves': 2780, 'max_depth': 7, 'min_child_samples': 84, 'subsample': 0.8244748440929208, 'colsample_bytree': 0.9680128454029922, 'reg_alpha': 2.5748287531875057e-05, 'reg_lambda': 0.0005823770716094656}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:16:41,213] Trial 97 finished with value: 0.46265235657145176 and parameters: {'boosting_type': 'goss', 'scale_pos_weight': 6.1207221998215475, 'n_estimators': 900, 'learning_rate': 0.025763868736120986, 'num_leaves': 2920, 'max_depth': 6, 'min_child_samples': 89, 'subsample': 0.8071544800179492, 'colsample_bytree': 0.6153400016666275, 'reg_alpha': 3.7302988273883673e-06, 'reg_lambda': 9.660127534558319e-05}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:18:18,586] Trial 98 finished with value: 0.4395987467531987 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 7.885217062977844, 'n_estimators': 1300, 'learning_rate': 0.033845768340927326, 'num_leaves': 2820, 'max_depth': 6, 'min_child_samples': 71, 'subsample': 0.6704897580341763, 'colsample_bytree': 0.6023197334904976, 'reg_alpha': 0.0019361947753674018, 'reg_lambda': 0.0014450763938484194}. Best is trial 11 with value: 0.4817346018383605.
[I 2025-09-21 01:19:48,254] Trial 99 finished with value: 0.4612496271199945 and parameters: {'boosting_type': 'dart', 'scale_pos_weight': 6.689297640925789, 'n_estimators': 500, 'learning_rate': 0.010515134130077456, 'num_leaves': 2660, 'max_depth': 12, 'min_child_samples': 48, 'subsample': 0.7611297070838973, 'colsample_bytree': 0.991492688228316, 'reg_alpha': 1.9967943493643786e-06, 'reg_lambda': 0.0024728294982792063}. Best is trial 11 with value: 0.4817346018383605.

--- Hyperparameter Optimization Complete ---

Best trial:
  Value: 0.4817346018383605
  Params: 
    boosting_type: dart
    scale_pos_weight: 7.009971471773609
    n_estimators: 200
    learning_rate: 0.011933663386885873
    num_leaves: 2980
    max_depth: 12
    min_child_samples: 99
    subsample: 0.976117178865086
    colsample_bytree: 0.6045884605959665
    reg_alpha: 1.0023526200624856e-08
    reg_lambda: 1.246535480598537e-08

# --- Hyperparameter Tuning with Optuna for XGBoost ---
def objective_xgb(trial):
    """
    This is the objective function that Optuna will optimize.
    A 'trial' is a single run of the model with a specific set of hyperparameters.
    """

    scale_pos_weight_value = y_train.value_counts()[0] / y_train.value_counts()[1]

    params = {
        'n_estimators': trial.suggest_int('n_estimators', 200, 2000, step=100),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),
        'max_depth': trial.suggest_int('max_depth', 3, 12),
        'grow_policy': trial.suggest_categorical('grow_policy', ['depthwise', 'lossguide']),
        'max_leaves': trial.suggest_int('max_leaves', 20, 3000, step=20),
        'subsample': trial.suggest_float('subsample', 0.7, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.7, 1.0),
        'gamma': trial.suggest_float('gamma', 1e-8, 1.0, log=True),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
        'scale_pos_weight': trial.suggest_float('scale_pos_weight', 
                                                scale_pos_weight_value * 0.8, 
                                                scale_pos_weight_value * 1.2),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 20),
        'eval_metric': 'logloss',
        'random_state': seed
    }

    pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor_engineered),
        ('classifier', XGBClassifier(**params))
    ])

    cv_strategy = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
    score = cross_val_score(
        pipeline, X_train, y_train, cv=cv_strategy, scoring='f1', n_jobs=-1
    ).mean()
    return score

study_xgb = optuna.create_study(direction='maximize', study_name='XGBoost Classifier Optimization')
print("\n--- Starting Hyperparameter Optimization for XGBoost Classifier ---\n")
study_xgb.optimize(objective_xgb, n_trials=100, show_progress_bar=True)
print("\n--- Hyperparameter Optimization Complete ---\n")
print("Best trial:")
trial = study_xgb.best_trial
print(f"  Value: {trial.value}")
print("  Params: ")
for key, value in trial.params.items():
    print(f"    {key}: {value}")

[I 2025-09-21 01:19:48,295] A new study created in memory with name: XGBoost Classifier Optimization

--- Starting Hyperparameter Optimization for XGBoost Classifier ---

  0%|          | 0/100 [00:00<?, ?it/s]

[I 2025-09-21 01:19:53,745] Trial 0 finished with value: 0.3956601734352805 and parameters: {'n_estimators': 1600, 'learning_rate': 0.2624297471767768, 'max_depth': 4, 'grow_policy': 'lossguide', 'max_leaves': 900, 'subsample': 0.7891813509393588, 'colsample_bytree': 0.7873553398135891, 'gamma': 0.0013284730452642528, 'reg_alpha': 0.000697255372442138, 'reg_lambda': 2.611956027021814e-06, 'scale_pos_weight': 8.539258670630973, 'min_child_weight': 5}. Best is trial 0 with value: 0.3956601734352805.
[I 2025-09-21 01:19:59,905] Trial 1 finished with value: 0.4350023925423942 and parameters: {'n_estimators': 800, 'learning_rate': 0.048045065422394805, 'max_depth': 12, 'grow_policy': 'depthwise', 'max_leaves': 2940, 'subsample': 0.7809113425872134, 'colsample_bytree': 0.8951515816843544, 'gamma': 0.00029374743176846535, 'reg_alpha': 0.45742668205018694, 'reg_lambda': 0.003083380161591768, 'scale_pos_weight': 8.21327151635441, 'min_child_weight': 17}. Best is trial 1 with value: 0.4350023925423942.
[I 2025-09-21 01:20:04,234] Trial 2 finished with value: 0.4009004140965956 and parameters: {'n_estimators': 600, 'learning_rate': 0.2060684971874041, 'max_depth': 12, 'grow_policy': 'lossguide', 'max_leaves': 2300, 'subsample': 0.8855440491284724, 'colsample_bytree': 0.8414661528366998, 'gamma': 0.011836407233325543, 'reg_alpha': 3.450897260249695e-06, 'reg_lambda': 0.01231755783003153, 'scale_pos_weight': 7.815629768118912, 'min_child_weight': 9}. Best is trial 1 with value: 0.4350023925423942.
[I 2025-09-21 01:20:05,372] Trial 3 finished with value: 0.40298409845823535 and parameters: {'n_estimators': 200, 'learning_rate': 0.29119149236669245, 'max_depth': 6, 'grow_policy': 'lossguide', 'max_leaves': 2120, 'subsample': 0.8283743780538109, 'colsample_bytree': 0.7717489791472064, 'gamma': 0.0006157360032220661, 'reg_alpha': 0.4232995930909043, 'reg_lambda': 0.05320618555211715, 'scale_pos_weight': 8.120947046519927, 'min_child_weight': 1}. Best is trial 1 with value: 0.4350023925423942.
[I 2025-09-21 01:20:14,249] Trial 4 finished with value: 0.3956461104521049 and parameters: {'n_estimators': 1600, 'learning_rate': 0.11474505191356626, 'max_depth': 8, 'grow_policy': 'lossguide', 'max_leaves': 1160, 'subsample': 0.9064879763486632, 'colsample_bytree': 0.7871637002547779, 'gamma': 0.00014895847448646044, 'reg_alpha': 0.0019970450891274192, 'reg_lambda': 1.1516867473071486, 'scale_pos_weight': 7.644253253990246, 'min_child_weight': 1}. Best is trial 1 with value: 0.4350023925423942.
[I 2025-09-21 01:20:19,266] Trial 5 finished with value: 0.4266138519103679 and parameters: {'n_estimators': 1400, 'learning_rate': 0.08301242862640172, 'max_depth': 8, 'grow_policy': 'lossguide', 'max_leaves': 1780, 'subsample': 0.9667168514254658, 'colsample_bytree': 0.7777339185561757, 'gamma': 0.42197257265158644, 'reg_alpha': 8.392152916933085e-07, 'reg_lambda': 5.623602560548156e-05, 'scale_pos_weight': 9.032488819917837, 'min_child_weight': 18}. Best is trial 1 with value: 0.4350023925423942.
[I 2025-09-21 01:20:25,380] Trial 6 finished with value: 0.38550123590757723 and parameters: {'n_estimators': 1300, 'learning_rate': 0.24109907429269417, 'max_depth': 7, 'grow_policy': 'lossguide', 'max_leaves': 660, 'subsample': 0.7090632474711908, 'colsample_bytree': 0.7656429343225785, 'gamma': 2.623247085599624e-08, 'reg_alpha': 0.00039101591206888345, 'reg_lambda': 0.0011659204281351213, 'scale_pos_weight': 6.667545305246668, 'min_child_weight': 5}. Best is trial 1 with value: 0.4350023925423942.
[I 2025-09-21 01:20:31,252] Trial 7 finished with value: 0.46702227717952993 and parameters: {'n_estimators': 1400, 'learning_rate': 0.014389779740163425, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 580, 'subsample': 0.8625401526638675, 'colsample_bytree': 0.8492682209699003, 'gamma': 1.5121874290264166e-06, 'reg_alpha': 2.973629356878654e-05, 'reg_lambda': 0.00019530231641678514, 'scale_pos_weight': 6.403552306664049, 'min_child_weight': 7}. Best is trial 7 with value: 0.46702227717952993.
[I 2025-09-21 01:20:33,068] Trial 8 finished with value: 0.4064350660267852 and parameters: {'n_estimators': 200, 'learning_rate': 0.24415561506416447, 'max_depth': 12, 'grow_policy': 'lossguide', 'max_leaves': 1400, 'subsample': 0.8519333746448005, 'colsample_bytree': 0.7008838541944229, 'gamma': 4.0231501425748675e-05, 'reg_alpha': 3.7742863889662327e-06, 'reg_lambda': 1.3372878678380764, 'scale_pos_weight': 6.780984140729642, 'min_child_weight': 7}. Best is trial 7 with value: 0.46702227717952993.
[I 2025-09-21 01:20:39,180] Trial 9 finished with value: 0.3848679916535106 and parameters: {'n_estimators': 1900, 'learning_rate': 0.2749107477652566, 'max_depth': 4, 'grow_policy': 'lossguide', 'max_leaves': 2980, 'subsample': 0.7017369663110757, 'colsample_bytree': 0.8564866641975625, 'gamma': 2.869399065611276e-05, 'reg_alpha': 0.3999324474599071, 'reg_lambda': 0.03426672022699508, 'scale_pos_weight': 7.942622071613496, 'min_child_weight': 6}. Best is trial 7 with value: 0.46702227717952993.
[I 2025-09-21 01:20:42,752] Trial 10 finished with value: 0.4670951541062996 and parameters: {'n_estimators': 1000, 'learning_rate': 0.013340347177114872, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 20, 'subsample': 0.9888995738210176, 'colsample_bytree': 0.9952405079984653, 'gamma': 7.13612690230233e-08, 'reg_alpha': 1.8870222498975557e-07, 'reg_lambda': 2.226646248972585e-06, 'scale_pos_weight': 6.161407351648147, 'min_child_weight': 13}. Best is trial 10 with value: 0.4670951541062996.
[I 2025-09-21 01:20:47,243] Trial 11 finished with value: 0.4721985383128831 and parameters: {'n_estimators': 1000, 'learning_rate': 0.011355152628552112, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 40, 'subsample': 0.9848853226271926, 'colsample_bytree': 0.9519090232352411, 'gamma': 2.1464230380674296e-07, 'reg_alpha': 2.4450802210993295e-08, 'reg_lambda': 1.3021330539428971e-08, 'scale_pos_weight': 6.063548875106149, 'min_child_weight': 13}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:20:50,358] Trial 12 finished with value: 0.4283816905216707 and parameters: {'n_estimators': 900, 'learning_rate': 0.1474689795242676, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 20, 'subsample': 0.9959243856261417, 'colsample_bytree': 0.9927404898782771, 'gamma': 1.1689922352181052e-08, 'reg_alpha': 1.1823136271648485e-08, 'reg_lambda': 1.1096465394798994e-08, 'scale_pos_weight': 6.038021180989774, 'min_child_weight': 14}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:20:55,141] Trial 13 finished with value: 0.4570471954588811 and parameters: {'n_estimators': 1000, 'learning_rate': 0.01745469181896479, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 60, 'subsample': 0.939235780938946, 'colsample_bytree': 0.9944867805267923, 'gamma': 3.802947189144446e-07, 'reg_alpha': 1.779232536532766e-08, 'reg_lambda': 1.0504590297305145e-08, 'scale_pos_weight': 7.1870499889076225, 'min_child_weight': 13}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:20:58,442] Trial 14 finished with value: 0.44329841443195317 and parameters: {'n_estimators': 600, 'learning_rate': 0.060942067478075686, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 320, 'subsample': 0.9930541847386836, 'colsample_bytree': 0.9401246793375212, 'gamma': 4.4162808404438977e-07, 'reg_alpha': 2.0911476861406385e-07, 'reg_lambda': 4.891469796716258e-07, 'scale_pos_weight': 6.202188156145131, 'min_child_weight': 13}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:21:04,368] Trial 15 finished with value: 0.41041362987975905 and parameters: {'n_estimators': 1200, 'learning_rate': 0.1011471631209664, 'max_depth': 9, 'grow_policy': 'depthwise', 'max_leaves': 440, 'subsample': 0.9345526090808652, 'colsample_bytree': 0.9494653901150821, 'gamma': 3.620645977150077e-06, 'reg_alpha': 8.881649897314521e-08, 'reg_lambda': 5.192913786914657e-07, 'scale_pos_weight': 7.143263626663894, 'min_child_weight': 20}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:21:08,045] Trial 16 finished with value: 0.4059477473505707 and parameters: {'n_estimators': 600, 'learning_rate': 0.17090858224932803, 'max_depth': 11, 'grow_policy': 'depthwise', 'max_leaves': 880, 'subsample': 0.9490459560298391, 'colsample_bytree': 0.9494564618626963, 'gamma': 8.196826798480716e-08, 'reg_alpha': 1.952848943677394e-05, 'reg_lambda': 9.033653300347747e-06, 'scale_pos_weight': 6.544543865330423, 'min_child_weight': 11}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:21:13,714] Trial 17 finished with value: 0.43094732124863483 and parameters: {'n_estimators': 1100, 'learning_rate': 0.050001015798506666, 'max_depth': 9, 'grow_policy': 'depthwise', 'max_leaves': 220, 'subsample': 0.907878080710718, 'colsample_bytree': 0.9244159118750706, 'gamma': 3.956401011262829e-06, 'reg_alpha': 0.009953668398644745, 'reg_lambda': 1.721826638511389e-07, 'scale_pos_weight': 6.972245200630843, 'min_child_weight': 16}. Best is trial 11 with value: 0.4721985383128831.
[I 2025-09-21 01:21:16,890] Trial 18 finished with value: 0.47286118661840604 and parameters: {'n_estimators': 800, 'learning_rate': 0.011686809446118235, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 1020, 'subsample': 0.9731220871001452, 'colsample_bytree': 0.973909882876838, 'gamma': 2.3727577227542704e-07, 'reg_alpha': 1.2599436855801167e-07, 'reg_lambda': 5.5112962177543836e-08, 'scale_pos_weight': 6.321077508088947, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:19,356] Trial 19 finished with value: 0.43209646633723614 and parameters: {'n_estimators': 700, 'learning_rate': 0.1397322721773884, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1680, 'subsample': 0.9642943535449471, 'colsample_bytree': 0.8990014589102091, 'gamma': 1.1008749530704962e-05, 'reg_alpha': 4.016644521088698e-05, 'reg_lambda': 6.28817673153782e-08, 'scale_pos_weight': 7.392458186210034, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:21,425] Trial 20 finished with value: 0.4576009627985579 and parameters: {'n_estimators': 500, 'learning_rate': 0.06580914137962618, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 1220, 'subsample': 0.9020277221878428, 'colsample_bytree': 0.9631536766771939, 'gamma': 3.7566005186421934e-07, 'reg_alpha': 5.492189270601441e-08, 'reg_lambda': 1.737207434009201e-05, 'scale_pos_weight': 6.398989916241454, 'min_child_weight': 15}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:26,366] Trial 21 finished with value: 0.4490376427820248 and parameters: {'n_estimators': 1000, 'learning_rate': 0.028347378169114222, 'max_depth': 9, 'grow_policy': 'depthwise', 'max_leaves': 740, 'subsample': 0.9996746273122197, 'colsample_bytree': 0.9798452133045081, 'gamma': 7.655715210623012e-08, 'reg_alpha': 4.373626546729899e-07, 'reg_lambda': 5.8087798473238716e-08, 'scale_pos_weight': 6.128371760006628, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:28,649] Trial 22 finished with value: 0.45503427079761494 and parameters: {'n_estimators': 900, 'learning_rate': 0.03459012921907055, 'max_depth': 3, 'grow_policy': 'depthwise', 'max_leaves': 380, 'subsample': 0.9713784418779003, 'colsample_bytree': 0.913069096654753, 'gamma': 8.599124552600456e-08, 'reg_alpha': 1.5761383215837595e-06, 'reg_lambda': 1.7522428521409166e-06, 'scale_pos_weight': 6.32298751395435, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:30,529] Trial 23 finished with value: 0.45137279589769175 and parameters: {'n_estimators': 400, 'learning_rate': 0.01219575445915509, 'max_depth': 11, 'grow_policy': 'depthwise', 'max_leaves': 20, 'subsample': 0.9222634908450074, 'colsample_bytree': 0.9811219603310557, 'gamma': 8.568649435667538e-07, 'reg_alpha': 1.017148197325688e-08, 'reg_lambda': 6.424203671572049e-08, 'scale_pos_weight': 6.83351276273573, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:33,867] Trial 24 finished with value: 0.44030515808916804 and parameters: {'n_estimators': 800, 'learning_rate': 0.08395511809425452, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 1040, 'subsample': 0.9722289591931532, 'colsample_bytree': 0.8738050123749223, 'gamma': 1.0623480074793728e-08, 'reg_alpha': 1.7848625774801006e-07, 'reg_lambda': 2.4432061327389807e-06, 'scale_pos_weight': 6.527733883116233, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:40,676] Trial 25 finished with value: 0.4341686113597665 and parameters: {'n_estimators': 1100, 'learning_rate': 0.04355154584341539, 'max_depth': 11, 'grow_policy': 'depthwise', 'max_leaves': 520, 'subsample': 0.9496691379751449, 'colsample_bytree': 0.9654476511964802, 'gamma': 8.565765057344594e-08, 'reg_alpha': 5.3863830054456665e-08, 'reg_lambda': 2.592796376525039e-07, 'scale_pos_weight': 6.078607624072567, 'min_child_weight': 15}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:42,623] Trial 26 finished with value: 0.4577800326526743 and parameters: {'n_estimators': 400, 'learning_rate': 0.07807354565630689, 'max_depth': 8, 'grow_policy': 'depthwise', 'max_leaves': 280, 'subsample': 0.9765950130357708, 'colsample_bytree': 0.9250108310232509, 'gamma': 5.583226646834095e-06, 'reg_alpha': 8.415573408201746e-06, 'reg_lambda': 1.3223883751599003e-08, 'scale_pos_weight': 6.267714165433298, 'min_child_weight': 19}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:48,372] Trial 27 finished with value: 0.4058968033941822 and parameters: {'n_estimators': 1200, 'learning_rate': 0.114650209735317, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 2500, 'subsample': 0.8813482579299897, 'colsample_bytree': 0.9995086735354837, 'gamma': 1.9656628742422978e-07, 'reg_alpha': 0.022541592647566677, 'reg_lambda': 1.3271004930541502e-05, 'scale_pos_weight': 6.630657441204735, 'min_child_weight': 8}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:21:51,629] Trial 28 finished with value: 0.4265170959844589 and parameters: {'n_estimators': 800, 'learning_rate': 0.18267774318906674, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1460, 'subsample': 0.8256604163394893, 'colsample_bytree': 0.968169272202017, 'gamma': 1.5560978516658153e-06, 'reg_alpha': 8.32160888082342, 'reg_lambda': 5.920986126172568e-08, 'scale_pos_weight': 6.965861138313108, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:01,176] Trial 29 finished with value: 0.42862659089487665 and parameters: {'n_estimators': 1600, 'learning_rate': 0.02693990275229177, 'max_depth': 9, 'grow_policy': 'depthwise', 'max_leaves': 800, 'subsample': 0.7681243789662164, 'colsample_bytree': 0.8132399058014343, 'gamma': 0.0038937121765521897, 'reg_alpha': 0.00013998905697939313, 'reg_lambda': 7.235487879065703e-07, 'scale_pos_weight': 8.906431015195949, 'min_child_weight': 4}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:11,624] Trial 30 finished with value: 0.41144191967675015 and parameters: {'n_estimators': 2000, 'learning_rate': 0.0567960614216361, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 1980, 'subsample': 0.9272485051841538, 'colsample_bytree': 0.942734943643413, 'gamma': 0.2021503955881055, 'reg_alpha': 6.287840112419495e-07, 'reg_lambda': 3.6775645590455543e-06, 'scale_pos_weight': 8.603499937025518, 'min_child_weight': 14}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:18,170] Trial 31 finished with value: 0.46753759012614965 and parameters: {'n_estimators': 1500, 'learning_rate': 0.010060054526136179, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 580, 'subsample': 0.8540447204598369, 'colsample_bytree': 0.8186910349325953, 'gamma': 1.180395655092258e-06, 'reg_alpha': 8.167948533976905e-05, 'reg_lambda': 0.00022513450887297142, 'scale_pos_weight': 6.409596471560795, 'min_child_weight': 8}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:24,811] Trial 32 finished with value: 0.4523771182733511 and parameters: {'n_estimators': 1800, 'learning_rate': 0.03334987487568941, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1020, 'subsample': 0.7336324939485714, 'colsample_bytree': 0.820620835626941, 'gamma': 5.624189452281812e-08, 'reg_alpha': 4.7905174629871216e-08, 'reg_lambda': 0.00010514581431018151, 'scale_pos_weight': 6.444317449176975, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:30,644] Trial 33 finished with value: 0.46426812556630226 and parameters: {'n_estimators': 1500, 'learning_rate': 0.015290644542409469, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 180, 'subsample': 0.8142110048328323, 'colsample_bytree': 0.7330244287427109, 'gamma': 2.651089423267157e-07, 'reg_alpha': 0.0014568169311993087, 'reg_lambda': 0.0010391565386663418, 'scale_pos_weight': 6.269785846579521, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:35,159] Trial 34 finished with value: 0.4525715306647765 and parameters: {'n_estimators': 1000, 'learning_rate': 0.04138884617772903, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 560, 'subsample': 0.7902694252363601, 'colsample_bytree': 0.8117292504063283, 'gamma': 2.567069374107981e-08, 'reg_alpha': 9.722479767014516e-05, 'reg_lambda': 9.73215372588186, 'scale_pos_weight': 6.70791851606854, 'min_child_weight': 17}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:40,447] Trial 35 finished with value: 0.4210682869026778 and parameters: {'n_estimators': 1200, 'learning_rate': 0.06974991122982335, 'max_depth': 8, 'grow_policy': 'depthwise', 'max_leaves': 200, 'subsample': 0.9825195324947731, 'colsample_bytree': 0.8792352524995425, 'gamma': 8.747289910805714e-07, 'reg_alpha': 1.841510830090174e-06, 'reg_lambda': 3.8897594262766156e-05, 'scale_pos_weight': 6.215544559335179, 'min_child_weight': 8}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:42,535] Trial 36 finished with value: 0.4546673202884544 and parameters: {'n_estimators': 700, 'learning_rate': 0.09645258422716683, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 1220, 'subsample': 0.9541381086844136, 'colsample_bytree': 0.8273439793388974, 'gamma': 2.073921766424935e-05, 'reg_alpha': 1.9675840996723062e-07, 'reg_lambda': 1.3007728856666215e-07, 'scale_pos_weight': 6.037279236670865, 'min_child_weight': 13}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:49,501] Trial 37 finished with value: 0.4299740060546323 and parameters: {'n_estimators': 1700, 'learning_rate': 0.04724144818182294, 'max_depth': 6, 'grow_policy': 'lossguide', 'max_leaves': 940, 'subsample': 0.8823477252797588, 'colsample_bytree': 0.7947306235848632, 'gamma': 0.00012330792407286732, 'reg_alpha': 5.017360057711381e-06, 'reg_lambda': 0.007991136930868996, 'scale_pos_weight': 7.475929353074779, 'min_child_weight': 3}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:22:59,435] Trial 38 finished with value: 0.4611468172533715 and parameters: {'n_estimators': 1400, 'learning_rate': 0.01051718549910496, 'max_depth': 12, 'grow_policy': 'depthwise', 'max_leaves': 700, 'subsample': 0.7588724588332463, 'colsample_bytree': 0.7586440799427097, 'gamma': 2.555967062796143e-08, 'reg_alpha': 3.58584960528681e-08, 'reg_lambda': 0.0007085187463354741, 'scale_pos_weight': 6.5119699194316585, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:03,884] Trial 39 finished with value: 0.4564788364584425 and parameters: {'n_estimators': 900, 'learning_rate': 0.027976620843367432, 'max_depth': 8, 'grow_policy': 'lossguide', 'max_leaves': 460, 'subsample': 0.9160492010213718, 'colsample_bytree': 0.8876545957331821, 'gamma': 0.0007258692023968945, 'reg_alpha': 4.330372313342727e-07, 'reg_lambda': 2.929429138142026e-08, 'scale_pos_weight': 6.896907288791993, 'min_child_weight': 8}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:09,802] Trial 40 finished with value: 0.40368775539065 and parameters: {'n_estimators': 1300, 'learning_rate': 0.12033027488899149, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 600, 'subsample': 0.8593094114446194, 'colsample_bytree': 0.9760656227036316, 'gamma': 0.04736102253192638, 'reg_alpha': 0.020022516080719865, 'reg_lambda': 1.1884100416588608e-06, 'scale_pos_weight': 8.2591602609512, 'min_child_weight': 6}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:16,233] Trial 41 finished with value: 0.46746908003991744 and parameters: {'n_estimators': 1500, 'learning_rate': 0.012222476804671317, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 120, 'subsample': 0.8611705406019051, 'colsample_bytree': 0.8602610660068366, 'gamma': 1.7233746687174144e-06, 'reg_alpha': 4.796166013320715e-05, 'reg_lambda': 0.0001812415186502692, 'scale_pos_weight': 6.3859962421866605, 'min_child_weight': 7}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:22,042] Trial 42 finished with value: 0.45561775227067897 and parameters: {'n_estimators': 1500, 'learning_rate': 0.027246690220990213, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 100, 'subsample': 0.839681680330607, 'colsample_bytree': 0.8587984731384983, 'gamma': 1.7919836208809084e-06, 'reg_alpha': 0.0008696892922272127, 'reg_lambda': 0.0003796633800867296, 'scale_pos_weight': 6.310957312697914, 'min_child_weight': 6}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:27,793] Trial 43 finished with value: 0.4681266395276792 and parameters: {'n_estimators': 1300, 'learning_rate': 0.010776139859974544, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 280, 'subsample': 0.8003532725146103, 'colsample_bytree': 0.8341657624822519, 'gamma': 1.7388347867077588e-07, 'reg_alpha': 0.004061722713453332, 'reg_lambda': 0.0043939639946763065, 'scale_pos_weight': 6.393459531418241, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:34,670] Trial 44 finished with value: 0.42708866728166706 and parameters: {'n_estimators': 1500, 'learning_rate': 0.051616781010905874, 'max_depth': 7, 'grow_policy': 'lossguide', 'max_leaves': 400, 'subsample': 0.8025830012748446, 'colsample_bytree': 0.8368075675115882, 'gamma': 1.1000182447439207e-05, 'reg_alpha': 0.00735250406850835, 'reg_lambda': 0.0031265858340670484, 'scale_pos_weight': 6.6374824342440535, 'min_child_weight': 7}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:40,979] Trial 45 finished with value: 0.4338636367411107 and parameters: {'n_estimators': 1300, 'learning_rate': 0.03893344023723125, 'max_depth': 8, 'grow_policy': 'depthwise', 'max_leaves': 280, 'subsample': 0.8699038803266742, 'colsample_bytree': 0.8620656272728925, 'gamma': 1.4809500773896176e-07, 'reg_alpha': 0.12918293230578307, 'reg_lambda': 0.03624338961354784, 'scale_pos_weight': 7.125900561720998, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:48,901] Trial 46 finished with value: 0.38650120004659094 and parameters: {'n_estimators': 1700, 'learning_rate': 0.23723806195643393, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 2660, 'subsample': 0.8412421781849209, 'colsample_bytree': 0.7987451264571719, 'gamma': 8.257617801314926e-07, 'reg_alpha': 0.00028107988868465716, 'reg_lambda': 0.10103208893118533, 'scale_pos_weight': 6.403005238547947, 'min_child_weight': 5}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:23:53,772] Trial 47 finished with value: 0.4376501128845126 and parameters: {'n_estimators': 1400, 'learning_rate': 0.023238758326542236, 'max_depth': 5, 'grow_policy': 'lossguide', 'max_leaves': 140, 'subsample': 0.7306390768246971, 'colsample_bytree': 0.8399650946245735, 'gamma': 5.510383847152904e-05, 'reg_alpha': 0.0029094624423851405, 'reg_lambda': 0.1380117265697961, 'scale_pos_weight': 7.80990676080532, 'min_child_weight': 8}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:00,642] Trial 48 finished with value: 0.38067103915559325 and parameters: {'n_estimators': 1700, 'learning_rate': 0.2948051251441237, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 640, 'subsample': 0.8965054420621014, 'colsample_bytree': 0.7767085980739266, 'gamma': 2.9931790658338037e-06, 'reg_alpha': 1.8378769330061776e-05, 'reg_lambda': 0.003902506680444907, 'scale_pos_weight': 6.742531151836608, 'min_child_weight': 3}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:06,222] Trial 49 finished with value: 0.468715852095964 and parameters: {'n_estimators': 1300, 'learning_rate': 0.011437627808856717, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 820, 'subsample': 0.8188812809121016, 'colsample_bytree': 0.7562850894505788, 'gamma': 6.156700308588432e-07, 'reg_alpha': 0.08765593140092563, 'reg_lambda': 0.012137070576150414, 'scale_pos_weight': 6.592724935525747, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:11,688] Trial 50 finished with value: 0.42167521829977545 and parameters: {'n_estimators': 1100, 'learning_rate': 0.0737071228042722, 'max_depth': 8, 'grow_policy': 'depthwise', 'max_leaves': 1340, 'subsample': 0.7963638409212802, 'colsample_bytree': 0.7010210062470027, 'gamma': 2.794860150765541e-08, 'reg_alpha': 0.09797137291021991, 'reg_lambda': 0.3684785875728814, 'scale_pos_weight': 7.286698960251636, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:17,330] Trial 51 finished with value: 0.46625584016883376 and parameters: {'n_estimators': 1300, 'learning_rate': 0.010899949891937807, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 840, 'subsample': 0.8142833417257275, 'colsample_bytree': 0.747920835804275, 'gamma': 5.191284100964493e-07, 'reg_alpha': 0.11450541642153864, 'reg_lambda': 0.019033692061469973, 'scale_pos_weight': 6.571019288890452, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:24,705] Trial 52 finished with value: 0.45584311303330843 and parameters: {'n_estimators': 1600, 'learning_rate': 0.02370563226317131, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 1640, 'subsample': 0.7790376722674306, 'colsample_bytree': 0.7237748386095746, 'gamma': 1.9042981554910066e-07, 'reg_alpha': 2.096790468840244, 'reg_lambda': 0.00032293581630267286, 'scale_pos_weight': 6.361317923981814, 'min_child_weight': 7}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:30,483] Trial 53 finished with value: 0.4357029740729928 and parameters: {'n_estimators': 1500, 'learning_rate': 0.05836947122307864, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 360, 'subsample': 0.8270209770474585, 'colsample_bytree': 0.8476608266729053, 'gamma': 1.8810461544542396e-06, 'reg_alpha': 7.596504744614933e-05, 'reg_lambda': 0.008949137009851194, 'scale_pos_weight': 6.48945450889333, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:36,238] Trial 54 finished with value: 0.4391135384370549 and parameters: {'n_estimators': 1200, 'learning_rate': 0.038007204400768234, 'max_depth': 8, 'grow_policy': 'depthwise', 'max_leaves': 480, 'subsample': 0.8436786381816813, 'colsample_bytree': 0.9040340117807453, 'gamma': 4.960374687577069e-07, 'reg_alpha': 0.0003973116787325565, 'reg_lambda': 0.0015954419091575565, 'scale_pos_weight': 6.174404634951303, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:40,001] Trial 55 finished with value: 0.4673203489668213 and parameters: {'n_estimators': 1100, 'learning_rate': 0.018865583614238655, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 740, 'subsample': 0.8128331551699082, 'colsample_bytree': 0.8705213150396982, 'gamma': 6.9374290071809226e-06, 'reg_alpha': 0.004539899381870043, 'reg_lambda': 0.0001401159435393064, 'scale_pos_weight': 6.187730925834343, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:46,465] Trial 56 finished with value: 0.3923124764982989 and parameters: {'n_estimators': 1400, 'learning_rate': 0.205750334150972, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 1080, 'subsample': 0.8701977601031847, 'colsample_bytree': 0.804435157272471, 'gamma': 1.6026149678641703e-07, 'reg_alpha': 0.02686136075920502, 'reg_lambda': 0.0022062754799085688, 'scale_pos_weight': 6.782867255362425, 'min_child_weight': 5}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:53,546] Trial 57 finished with value: 0.4350414381456898 and parameters: {'n_estimators': 1400, 'learning_rate': 0.033882947971222876, 'max_depth': 9, 'grow_policy': 'depthwise', 'max_leaves': 300, 'subsample': 0.8517726880908437, 'colsample_bytree': 0.8299322627610985, 'gamma': 2.752424458837501e-06, 'reg_alpha': 1.2288499925547345e-05, 'reg_lambda': 0.006525020526363483, 'scale_pos_weight': 6.392970566540893, 'min_child_weight': 14}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:24:58,525] Trial 58 finished with value: 0.46337545263268654 and parameters: {'n_estimators': 1300, 'learning_rate': 0.020604677799883564, 'max_depth': 6, 'grow_policy': 'lossguide', 'max_leaves': 940, 'subsample': 0.7763606632160366, 'colsample_bytree': 0.7909318841816504, 'gamma': 4.453691299781097e-08, 'reg_alpha': 0.4143979959655697, 'reg_lambda': 4.554256932327179e-05, 'scale_pos_weight': 6.1063978978070645, 'min_child_weight': 7}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:02,242] Trial 59 finished with value: 0.44746231167619593 and parameters: {'n_estimators': 900, 'learning_rate': 0.04717573350006052, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 140, 'subsample': 0.7569341922929818, 'colsample_bytree': 0.9279219950497724, 'gamma': 9.113910761860211e-07, 'reg_alpha': 2.3487684844920513e-08, 'reg_lambda': 0.019047528817143108, 'scale_pos_weight': 6.6664941852434545, 'min_child_weight': 6}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:09,589] Trial 60 finished with value: 0.4643886629013436 and parameters: {'n_estimators': 1600, 'learning_rate': 0.010574694789020024, 'max_depth': 8, 'grow_policy': 'depthwise', 'max_leaves': 540, 'subsample': 0.9589861561714048, 'colsample_bytree': 0.782063646960391, 'gamma': 3.376490031782709e-07, 'reg_alpha': 1.1512662867389858, 'reg_lambda': 0.0005997544123602966, 'scale_pos_weight': 7.064886763840816, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:12,486] Trial 61 finished with value: 0.45568036423725305 and parameters: {'n_estimators': 1100, 'learning_rate': 0.02152649988499803, 'max_depth': 3, 'grow_policy': 'depthwise', 'max_leaves': 720, 'subsample': 0.8098607685613605, 'colsample_bytree': 0.8882907408395891, 'gamma': 6.645604262072296e-06, 'reg_alpha': 0.0038815479893864486, 'reg_lambda': 0.0001383972330013142, 'scale_pos_weight': 6.241323288698982, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:15,428] Trial 62 finished with value: 0.46839440839078694 and parameters: {'n_estimators': 800, 'learning_rate': 0.019313392838321758, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 20, 'subsample': 0.8247418783157037, 'colsample_bytree': 0.8687529482602673, 'gamma': 1.7550596449295085e-05, 'reg_alpha': 0.0632646927569288, 'reg_lambda': 2.482692726193811e-08, 'scale_pos_weight': 6.167013454223086, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:18,314] Trial 63 finished with value: 0.4654957321754747 and parameters: {'n_estimators': 800, 'learning_rate': 0.033411619297150884, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 40, 'subsample': 0.8335382493191295, 'colsample_bytree': 0.9556788807129151, 'gamma': 2.323809453867427e-05, 'reg_alpha': 0.05110227104412802, 'reg_lambda': 2.9349311009652708e-08, 'scale_pos_weight': 6.322814968446206, 'min_child_weight': 13}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:20,521] Trial 64 finished with value: 0.4545017758720712 and parameters: {'n_estimators': 700, 'learning_rate': 0.0627385174172565, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 220, 'subsample': 0.8237910669810442, 'colsample_bytree': 0.8489753326670717, 'gamma': 1.1695097822089919e-07, 'reg_alpha': 0.001178201791774602, 'reg_lambda': 2.2386807180975017e-08, 'scale_pos_weight': 6.574822113285198, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:24,885] Trial 65 finished with value: 0.4648455554387894 and parameters: {'n_estimators': 1000, 'learning_rate': 0.021593673405400873, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 380, 'subsample': 0.9378897076286122, 'colsample_bytree': 0.9881681818250697, 'gamma': 1.4003173668292383e-05, 'reg_alpha': 4.590532750850407e-05, 'reg_lambda': 5.376769215857362e-06, 'scale_pos_weight': 6.037262975777624, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:28,881] Trial 66 finished with value: 0.4619745786895818 and parameters: {'n_estimators': 1200, 'learning_rate': 0.010042605989211443, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 20, 'subsample': 0.9872154903570327, 'colsample_bytree': 0.9126307233874188, 'gamma': 4.382620364821902e-05, 'reg_alpha': 0.2038040609266617, 'reg_lambda': 1.4589354154687385e-07, 'scale_pos_weight': 6.467137173605, 'min_child_weight': 8}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:31,609] Trial 67 finished with value: 0.4622812728648881 and parameters: {'n_estimators': 600, 'learning_rate': 0.04071822291726568, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 260, 'subsample': 0.8581366218151848, 'colsample_bytree': 0.7667133003996168, 'gamma': 8.533251333378186e-07, 'reg_alpha': 9.08875570437201e-08, 'reg_lambda': 3.0023673869558226e-07, 'scale_pos_weight': 6.165699460190818, 'min_child_weight': 14}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:33,738] Trial 68 finished with value: 0.45849391692856 and parameters: {'n_estimators': 500, 'learning_rate': 0.051767761356998405, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 180, 'subsample': 0.7911215387252797, 'colsample_bytree': 0.9321224225225626, 'gamma': 3.0208106466546446e-07, 'reg_alpha': 0.00022370783994002326, 'reg_lambda': 9.551744778498043e-08, 'scale_pos_weight': 6.279436417491205, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:38,290] Trial 69 finished with value: 0.45527036310102564 and parameters: {'n_estimators': 800, 'learning_rate': 0.030256401719270975, 'max_depth': 9, 'grow_policy': 'depthwise', 'max_leaves': 440, 'subsample': 0.799481027900815, 'colsample_bytree': 0.8186046290645246, 'gamma': 4.338748180425061e-06, 'reg_alpha': 0.009722930358099353, 'reg_lambda': 3.703262743702002e-08, 'scale_pos_weight': 6.871786102694593, 'min_child_weight': 15}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:41,494] Trial 70 finished with value: 0.43623481670410713 and parameters: {'n_estimators': 700, 'learning_rate': 0.08910557933295102, 'max_depth': 7, 'grow_policy': 'depthwise', 'max_leaves': 100, 'subsample': 0.8208548312311611, 'colsample_bytree': 0.8290010012828289, 'gamma': 0.00021226608896594644, 'reg_alpha': 1.1730929173956958e-07, 'reg_lambda': 1.2984363456607027e-08, 'scale_pos_weight': 6.137382089975778, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:45,375] Trial 71 finished with value: 0.4639169138918956 and parameters: {'n_estimators': 1100, 'learning_rate': 0.018828870765040133, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 820, 'subsample': 0.805834305851269, 'colsample_bytree': 0.86545293163394, 'gamma': 6.0508649962718955e-06, 'reg_alpha': 0.000592138164888535, 'reg_lambda': 2.166215810776986e-05, 'scale_pos_weight': 6.420169988803395, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:48,636] Trial 72 finished with value: 0.4682875898459443 and parameters: {'n_estimators': 900, 'learning_rate': 0.018804281369997837, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 660, 'subsample': 0.8346396094710209, 'colsample_bytree': 0.8750305669609326, 'gamma': 9.690963345799115e-06, 'reg_alpha': 0.005899877787472581, 'reg_lambda': 6.817260262638184e-05, 'scale_pos_weight': 6.207820091670942, 'min_child_weight': 13}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:51,448] Trial 73 finished with value: 0.4610395238529647 and parameters: {'n_estimators': 900, 'learning_rate': 0.030165563541452865, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 640, 'subsample': 0.8345919545985537, 'colsample_bytree': 0.8775246739043321, 'gamma': 6.456122288043649e-05, 'reg_alpha': 0.015529816412136434, 'reg_lambda': 0.00022225264976365378, 'scale_pos_weight': 6.244726975687677, 'min_child_weight': 14}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:55,243] Trial 74 finished with value: 0.45359199220779994 and parameters: {'n_estimators': 1000, 'learning_rate': 0.0426495567191255, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 540, 'subsample': 0.872415588435768, 'colsample_bytree': 0.8900541300826741, 'gamma': 1.181601952703112e-06, 'reg_alpha': 0.07145990516409133, 'reg_lambda': 7.274274663778442e-05, 'scale_pos_weight': 6.346464393971377, 'min_child_weight': 13}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:25:58,102] Trial 75 finished with value: 0.4708086804770522 and parameters: {'n_estimators': 800, 'learning_rate': 0.017333993883161012, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 300, 'subsample': 0.8483537800434028, 'colsample_bytree': 0.8534475787694001, 'gamma': 1.867393311571737e-06, 'reg_alpha': 0.24315606867552583, 'reg_lambda': 0.004886345005552461, 'scale_pos_weight': 6.119821217701377, 'min_child_weight': 16}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:01,181] Trial 76 finished with value: 0.46975877681540057 and parameters: {'n_estimators': 900, 'learning_rate': 0.01857290440341677, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1180, 'subsample': 0.8966753679010739, 'colsample_bytree': 0.8531375734431754, 'gamma': 1.0666241544073726e-05, 'reg_alpha': 0.047011577629263744, 'reg_lambda': 0.05222543655704763, 'scale_pos_weight': 6.113551851088764, 'min_child_weight': 16}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:03,728] Trial 77 finished with value: 0.4662865238370566 and parameters: {'n_estimators': 800, 'learning_rate': 0.019593786312666853, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 1100, 'subsample': 0.8330665382024941, 'colsample_bytree': 0.8537437495471645, 'gamma': 1.3827172663828849e-05, 'reg_alpha': 1.336526244447304, 'reg_lambda': 0.12540539139334897, 'scale_pos_weight': 6.118262595212759, 'min_child_weight': 17}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:06,311] Trial 78 finished with value: 0.4115974307953131 and parameters: {'n_estimators': 700, 'learning_rate': 0.2828221945706698, 'max_depth': 5, 'grow_policy': 'lossguide', 'max_leaves': 1400, 'subsample': 0.8941805614483918, 'colsample_bytree': 0.9052207216738841, 'gamma': 9.523414814116492e-05, 'reg_alpha': 0.24046882591399993, 'reg_lambda': 0.04551938121219238, 'scale_pos_weight': 6.047360697309243, 'min_child_weight': 16}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:09,198] Trial 79 finished with value: 0.456286444043786 and parameters: {'n_estimators': 900, 'learning_rate': 0.051832861661681565, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1240, 'subsample': 0.9777085147531986, 'colsample_bytree': 0.8444718491836767, 'gamma': 2.9696181691368173e-05, 'reg_alpha': 0.0464789134309619, 'reg_lambda': 0.4226212893022989, 'scale_pos_weight': 6.18689231361736, 'min_child_weight': 20}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:12,254] Trial 80 finished with value: 0.46115921002837457 and parameters: {'n_estimators': 1000, 'learning_rate': 0.027688184262092193, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 1020, 'subsample': 0.8428802959710316, 'colsample_bytree': 0.9549900871754409, 'gamma': 5.554270444252635e-07, 'reg_alpha': 0.0272858627393425, 'reg_lambda': 0.07640232874532955, 'scale_pos_weight': 6.253453409182176, 'min_child_weight': 16}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:15,153] Trial 81 finished with value: 0.47154214099633185 and parameters: {'n_estimators': 800, 'learning_rate': 0.016519068568841448, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 920, 'subsample': 0.8185302171227108, 'colsample_bytree': 0.8672535473929293, 'gamma': 2.7121753157987864e-06, 'reg_alpha': 0.7944984154384994, 'reg_lambda': 0.2836798057146191, 'scale_pos_weight': 6.118560357991541, 'min_child_weight': 18}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:18,006] Trial 82 finished with value: 0.46241123382747495 and parameters: {'n_estimators': 900, 'learning_rate': 0.03488155987351896, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 920, 'subsample': 0.9980132850698462, 'colsample_bytree': 0.8805060338650063, 'gamma': 9.247941959759321e-06, 'reg_alpha': 0.7951890254407002, 'reg_lambda': 1.7142922176243256, 'scale_pos_weight': 6.099105795300421, 'min_child_weight': 19}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:20,632] Trial 83 finished with value: 0.4628707432077861 and parameters: {'n_estimators': 800, 'learning_rate': 0.02112894336854903, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 1300, 'subsample': 0.8200796322391286, 'colsample_bytree': 0.8668372067823162, 'gamma': 2.7709450096111378e-06, 'reg_alpha': 3.291226596356691, 'reg_lambda': 0.6493958256477931, 'scale_pos_weight': 6.296689319487206, 'min_child_weight': 18}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:23,239] Trial 84 finished with value: 0.44350827323519904 and parameters: {'n_estimators': 700, 'learning_rate': 0.01816456278483899, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1120, 'subsample': 0.8470438323033062, 'colsample_bytree': 0.9750327636204056, 'gamma': 1.712732848358477e-05, 'reg_alpha': 0.24133612785928452, 'reg_lambda': 0.020549862589051553, 'scale_pos_weight': 7.635112937671198, 'min_child_weight': 18}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:26,595] Trial 85 finished with value: 0.4168048838320704 and parameters: {'n_estimators': 900, 'learning_rate': 0.16952431395352488, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 960, 'subsample': 0.9656000177503969, 'colsample_bytree': 0.8982266385336822, 'gamma': 2.2196145881384372e-06, 'reg_alpha': 0.5991532234542337, 'reg_lambda': 0.00425880892791125, 'scale_pos_weight': 6.12278229137877, 'min_child_weight': 17}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:28,726] Trial 86 finished with value: 0.41614331633780105 and parameters: {'n_estimators': 600, 'learning_rate': 0.2574203036662891, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1160, 'subsample': 0.9479061912634725, 'colsample_bytree': 0.8544039003639402, 'gamma': 1.0154754349995383e-07, 'reg_alpha': 0.0352286539617027, 'reg_lambda': 0.24175859260986643, 'scale_pos_weight': 6.494410386768356, 'min_child_weight': 16}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:34,718] Trial 87 finished with value: 0.4518260761269791 and parameters: {'n_estimators': 800, 'learning_rate': 0.039527568352622916, 'max_depth': 11, 'grow_policy': 'depthwise', 'max_leaves': 780, 'subsample': 0.7955304381960566, 'colsample_bytree': 0.839730348335598, 'gamma': 4.792886984289502e-08, 'reg_alpha': 4.745977920548776, 'reg_lambda': 1.617272362562416e-08, 'scale_pos_weight': 6.2202057682467125, 'min_child_weight': 15}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:36,819] Trial 88 finished with value: 0.46612006412582546 and parameters: {'n_estimators': 500, 'learning_rate': 0.02665311143200188, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 1020, 'subsample': 0.8194769234658767, 'colsample_bytree': 0.8821841333275224, 'gamma': 0.00040906720833846567, 'reg_alpha': 0.14623070652386919, 'reg_lambda': 0.011275089136953603, 'scale_pos_weight': 6.579149290784658, 'min_child_weight': 13}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:39,636] Trial 89 finished with value: 0.4509862547149141 and parameters: {'n_estimators': 1000, 'learning_rate': 0.014907002567533077, 'max_depth': 3, 'grow_policy': 'lossguide', 'max_leaves': 880, 'subsample': 0.787110916159299, 'colsample_bytree': 0.8352647989826142, 'gamma': 4.9994959847377555e-06, 'reg_alpha': 0.015558435527761151, 'reg_lambda': 0.025231346943725346, 'scale_pos_weight': 6.328207370747909, 'min_child_weight': 15}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:42,541] Trial 90 finished with value: 0.44520059665536166 and parameters: {'n_estimators': 900, 'learning_rate': 0.12978093714116817, 'max_depth': 4, 'grow_policy': 'depthwise', 'max_leaves': 1620, 'subsample': 0.8300966804140855, 'colsample_bytree': 0.869324113377387, 'gamma': 6.546485754790449e-07, 'reg_alpha': 0.002007597682193822, 'reg_lambda': 3.142493593979942, 'scale_pos_weight': 6.082284163044211, 'min_child_weight': 12}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:45,805] Trial 91 finished with value: 0.4713254084303665 and parameters: {'n_estimators': 800, 'learning_rate': 0.010275233705822806, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 660, 'subsample': 0.9134191519025113, 'colsample_bytree': 0.807472848781257, 'gamma': 2.5354796046260874e-07, 'reg_alpha': 0.006711631174959808, 'reg_lambda': 0.0007660113995522316, 'scale_pos_weight': 6.4358377867640915, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:48,550] Trial 92 finished with value: 0.4671076337708417 and parameters: {'n_estimators': 800, 'learning_rate': 0.016152278421170167, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 680, 'subsample': 0.9890673000669361, 'colsample_bytree': 0.8077357935085357, 'gamma': 2.41725188369714e-07, 'reg_alpha': 0.006229201673505779, 'reg_lambda': 0.005295879366983507, 'scale_pos_weight': 6.180317105689837, 'min_child_weight': 19}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:51,614] Trial 93 finished with value: 0.4617224665504634 and parameters: {'n_estimators': 800, 'learning_rate': 0.027991100350079894, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 340, 'subsample': 0.9078273365614062, 'colsample_bytree': 0.7433209500608039, 'gamma': 1.2085014174585651e-06, 'reg_alpha': 0.06895478125740317, 'reg_lambda': 0.0024790326647598174, 'scale_pos_weight': 6.445314677266651, 'min_child_weight': 9}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:54,076] Trial 94 finished with value: 0.46537332856035485 and parameters: {'n_estimators': 700, 'learning_rate': 0.03567825844292043, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 1880, 'subsample': 0.9221171620654783, 'colsample_bytree': 0.8533173112825077, 'gamma': 2.0513202234509699e-07, 'reg_alpha': 0.3468318498324927, 'reg_lambda': 0.0006219937844260975, 'scale_pos_weight': 6.359756557659078, 'min_child_weight': 11}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:57,448] Trial 95 finished with value: 0.43963127817471426 and parameters: {'n_estimators': 900, 'learning_rate': 0.015903761199556526, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 860, 'subsample': 0.8881594101220307, 'colsample_bytree': 0.8247542006099816, 'gamma': 3.658548377575938e-07, 'reg_alpha': 1.368756090555476e-08, 'reg_lambda': 0.001203277267020495, 'scale_pos_weight': 8.155635900523297, 'min_child_weight': 18}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:26:59,566] Trial 96 finished with value: 0.458988662249063 and parameters: {'n_estimators': 600, 'learning_rate': 0.046100898288497234, 'max_depth': 5, 'grow_policy': 'depthwise', 'max_leaves': 780, 'subsample': 0.8756246230289658, 'colsample_bytree': 0.8733115543254502, 'gamma': 8.892991633671773e-06, 'reg_alpha': 0.013126317280652389, 'reg_lambda': 0.07169991915358571, 'scale_pos_weight': 6.2608806786932, 'min_child_weight': 10}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:27:03,507] Trial 97 finished with value: 0.4611073409233125 and parameters: {'n_estimators': 1000, 'learning_rate': 0.02407180831200035, 'max_depth': 6, 'grow_policy': 'depthwise', 'max_leaves': 260, 'subsample': 0.8091779222972482, 'colsample_bytree': 0.8633319285449816, 'gamma': 1.7828091101546143e-08, 'reg_alpha': 0.002187260066346106, 'reg_lambda': 0.0015951514368829549, 'scale_pos_weight': 6.530981338264652, 'min_child_weight': 13}. Best is trial 18 with value: 0.47286118661840604.
[I 2025-09-21 01:27:08,494] Trial 98 finished with value: 0.4745618621068545 and parameters: {'n_estimators': 800, 'learning_rate': 0.010762488703917415, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 480, 'subsample': 0.8364716594307703, 'colsample_bytree': 0.8330690746752506, 'gamma': 6.498450169854745e-08, 'reg_alpha': 0.07515286012169026, 'reg_lambda': 5.7216934259911406e-08, 'scale_pos_weight': 6.045245907577975, 'min_child_weight': 9}. Best is trial 98 with value: 0.4745618621068545.
[I 2025-09-21 01:27:12,628] Trial 99 finished with value: 0.45327857003522903 and parameters: {'n_estimators': 700, 'learning_rate': 0.03199771983820424, 'max_depth': 10, 'grow_policy': 'depthwise', 'max_leaves': 640, 'subsample': 0.8645670247718279, 'colsample_bytree': 0.9880768816172728, 'gamma': 3.521468271195627e-08, 'reg_alpha': 0.06979062195171419, 'reg_lambda': 4.6840093449535884e-08, 'scale_pos_weight': 6.053372950819839, 'min_child_weight': 12}. Best is trial 98 with value: 0.4745618621068545.

--- Hyperparameter Optimization Complete ---

Best trial:
  Value: 0.4745618621068545
  Params: 
    n_estimators: 800
    learning_rate: 0.010762488703917415
    max_depth: 10
    grow_policy: depthwise
    max_leaves: 480
    subsample: 0.8364716594307703
    colsample_bytree: 0.8330690746752506
    gamma: 6.498450169854745e-08
    reg_alpha: 0.07515286012169026
    reg_lambda: 5.7216934259911406e-08
    scale_pos_weight: 6.045245907577975
    min_child_weight: 9

# --- Hyperparameter Tuning with Optuna for HistGradientBoostingClassifier ---
def objective_hgb(trial):
    """
    This is the objective function for tuning the HistGradientBoostingClassifier using Optuna.
    A 'trial' is a single run of the model with a specific set of hyperparameters.
    """

    params = {
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3),
        'max_iter': trial.suggest_int('max_iter', 100, 3000),
        'max_leaf_nodes': trial.suggest_int('max_leaf_nodes', 20, 3000, step=20),
        'max_depth': trial.suggest_int('max_depth', 3, 15),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 20, 200),
        'l2_regularization': trial.suggest_float('l2_regularization', 1e-8, 10.0, log=True),
        'max_bins': trial.suggest_int('max_bins', 100, 255),
        'class_weight': 'balanced',
        'random_state': seed
    }

    pipeline = Pipeline(steps=[
        ('preprocessor', preprocessor_engineered),
        ('classifier', HistGradientBoostingClassifier(**params))
    ])

    cv_strategy = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
    score = cross_val_score(
        pipeline, X_train, y_train, cv=cv_strategy, scoring='f1', n_jobs=-1
    ).mean()
    return score

study_hgb = optuna.create_study(direction='maximize', study_name='HistGradientBoosting Classifier Optimization')
print("\n--- Starting Hyperparameter Optimization for HistGradientBoosting Classifier ---\n")
study_hgb.optimize(objective_hgb, n_trials=100, show_progress_bar=True)
print("\n--- Hyperparameter Optimization Complete ---\n")
print("Best trial:")
trial = study_hgb.best_trial
print(f"  Value: {trial.value}")
print("  Params: ")
for key, value in trial.params.items():
    print(f"    {key}: {value}")

[I 2025-09-21 01:27:12,664] A new study created in memory with name: HistGradientBoosting Classifier Optimization

--- Starting Hyperparameter Optimization for HistGradientBoosting Classifier ---

  0%|          | 0/100 [00:00<?, ?it/s]

[I 2025-09-21 01:27:13,908] Trial 0 finished with value: 0.44069215194145545 and parameters: {'learning_rate': 0.17535474520640182, 'max_iter': 2127, 'max_leaf_nodes': 1420, 'max_depth': 15, 'min_samples_leaf': 179, 'l2_regularization': 0.9635695636390339, 'max_bins': 105}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:14,767] Trial 1 finished with value: 0.43258281453184544 and parameters: {'learning_rate': 0.26992229724119965, 'max_iter': 1505, 'max_leaf_nodes': 1600, 'max_depth': 15, 'min_samples_leaf': 143, 'l2_regularization': 2.105940346639055e-08, 'max_bins': 125}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:15,690] Trial 2 finished with value: 0.43361633938766947 and parameters: {'learning_rate': 0.23859445534942025, 'max_iter': 782, 'max_leaf_nodes': 720, 'max_depth': 13, 'min_samples_leaf': 53, 'l2_regularization': 0.11677803114625733, 'max_bins': 121}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:16,526] Trial 3 finished with value: 0.43015271577193726 and parameters: {'learning_rate': 0.24891653518679158, 'max_iter': 1460, 'max_leaf_nodes': 500, 'max_depth': 12, 'min_samples_leaf': 84, 'l2_regularization': 8.94153501841158e-07, 'max_bins': 213}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:18,961] Trial 4 finished with value: 0.4341532846189387 and parameters: {'learning_rate': 0.07345146909599311, 'max_iter': 2267, 'max_leaf_nodes': 1920, 'max_depth': 3, 'min_samples_leaf': 106, 'l2_regularization': 0.011042750804179957, 'max_bins': 196}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:19,991] Trial 5 finished with value: 0.4387778884699764 and parameters: {'learning_rate': 0.13182016046832745, 'max_iter': 1564, 'max_leaf_nodes': 1740, 'max_depth': 14, 'min_samples_leaf': 160, 'l2_regularization': 1.2854068128496587e-06, 'max_bins': 143}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:20,957] Trial 6 finished with value: 0.44049510336424624 and parameters: {'learning_rate': 0.15349353721982376, 'max_iter': 214, 'max_leaf_nodes': 2900, 'max_depth': 10, 'min_samples_leaf': 73, 'l2_regularization': 0.00118262398700253, 'max_bins': 254}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:21,867] Trial 7 finished with value: 0.4344234080309467 and parameters: {'learning_rate': 0.233092675085641, 'max_iter': 2033, 'max_leaf_nodes': 2480, 'max_depth': 15, 'min_samples_leaf': 193, 'l2_regularization': 0.025176303853173952, 'max_bins': 253}. Best is trial 0 with value: 0.44069215194145545.
[I 2025-09-21 01:27:24,352] Trial 8 finished with value: 0.44390366609440324 and parameters: {'learning_rate': 0.04873129882823929, 'max_iter': 2292, 'max_leaf_nodes': 740, 'max_depth': 14, 'min_samples_leaf': 178, 'l2_regularization': 0.009081048881757255, 'max_bins': 206}. Best is trial 8 with value: 0.44390366609440324.
[I 2025-09-21 01:27:25,261] Trial 9 finished with value: 0.4329136387037381 and parameters: {'learning_rate': 0.262818962432208, 'max_iter': 1351, 'max_leaf_nodes': 1560, 'max_depth': 13, 'min_samples_leaf': 194, 'l2_regularization': 2.4933504219011033, 'max_bins': 116}. Best is trial 8 with value: 0.44390366609440324.
[I 2025-09-21 01:27:29,494] Trial 10 finished with value: 0.44946582529690404 and parameters: {'learning_rate': 0.021792567418289804, 'max_iter': 2957, 'max_leaf_nodes': 180, 'max_depth': 7, 'min_samples_leaf': 133, 'l2_regularization': 9.48445015071177e-05, 'max_bins': 166}. Best is trial 10 with value: 0.44946582529690404.
[I 2025-09-21 01:27:37,106] Trial 11 finished with value: 0.4453539759134226 and parameters: {'learning_rate': 0.011486080122700443, 'max_iter': 2932, 'max_leaf_nodes': 140, 'max_depth': 6, 'min_samples_leaf': 137, 'l2_regularization': 5.1286737582853535e-05, 'max_bins': 170}. Best is trial 10 with value: 0.44946582529690404.
[I 2025-09-21 01:27:42,884] Trial 12 finished with value: 0.44602849494773256 and parameters: {'learning_rate': 0.01628989419514663, 'max_iter': 2993, 'max_leaf_nodes': 120, 'max_depth': 6, 'min_samples_leaf': 124, 'l2_regularization': 3.009148898026161e-05, 'max_bins': 164}. Best is trial 10 with value: 0.44946582529690404.
[I 2025-09-21 01:27:44,086] Trial 13 finished with value: 0.4442992147235968 and parameters: {'learning_rate': 0.09259875438692684, 'max_iter': 2874, 'max_leaf_nodes': 40, 'max_depth': 7, 'min_samples_leaf': 113, 'l2_regularization': 6.412157136814757e-05, 'max_bins': 160}. Best is trial 10 with value: 0.44946582529690404.
[I 2025-09-21 01:27:48,787] Trial 14 finished with value: 0.4420779789342378 and parameters: {'learning_rate': 0.02291472197275544, 'max_iter': 2668, 'max_leaf_nodes': 1160, 'max_depth': 5, 'min_samples_leaf': 120, 'l2_regularization': 6.59784042032243e-06, 'max_bins': 150}. Best is trial 10 with value: 0.44946582529690404.
[I 2025-09-21 01:27:49,984] Trial 15 finished with value: 0.4465455531520245 and parameters: {'learning_rate': 0.0992689671216345, 'max_iter': 2520, 'max_leaf_nodes': 400, 'max_depth': 8, 'min_samples_leaf': 20, 'l2_regularization': 0.00044867067002085745, 'max_bins': 182}. Best is trial 10 with value: 0.44946582529690404.
[I 2025-09-21 01:27:51,112] Trial 16 finished with value: 0.44966332375186335 and parameters: {'learning_rate': 0.11631412390732328, 'max_iter': 2585, 'max_leaf_nodes': 1040, 'max_depth': 9, 'min_samples_leaf': 30, 'l2_regularization': 0.0005631593916178514, 'max_bins': 190}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:51,999] Trial 17 finished with value: 0.43910778927605054 and parameters: {'learning_rate': 0.1911978384446661, 'max_iter': 2536, 'max_leaf_nodes': 1020, 'max_depth': 10, 'min_samples_leaf': 33, 'l2_regularization': 2.1345423143492195e-08, 'max_bins': 226}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:53,138] Trial 18 finished with value: 0.4474458504872085 and parameters: {'learning_rate': 0.12084803482311396, 'max_iter': 1708, 'max_leaf_nodes': 1000, 'max_depth': 9, 'min_samples_leaf': 75, 'l2_regularization': 0.0009118217032910826, 'max_bins': 188}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:55,485] Trial 19 finished with value: 0.43901162713579944 and parameters: {'learning_rate': 0.05976269401201817, 'max_iter': 1064, 'max_leaf_nodes': 2240, 'max_depth': 4, 'min_samples_leaf': 94, 'l2_regularization': 8.738387435836632e-07, 'max_bins': 230}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:56,411] Trial 20 finished with value: 0.44089376803529695 and parameters: {'learning_rate': 0.20027071067557622, 'max_iter': 2007, 'max_leaf_nodes': 580, 'max_depth': 11, 'min_samples_leaf': 53, 'l2_regularization': 0.22944405725062156, 'max_bins': 142}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:57,513] Trial 21 finished with value: 0.4449655186211786 and parameters: {'learning_rate': 0.12794924881055858, 'max_iter': 1749, 'max_leaf_nodes': 1040, 'max_depth': 9, 'min_samples_leaf': 61, 'l2_regularization': 0.0015817033491475104, 'max_bins': 186}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:58,615] Trial 22 finished with value: 0.44623468198591193 and parameters: {'learning_rate': 0.12565856459816144, 'max_iter': 2614, 'max_leaf_nodes': 1340, 'max_depth': 8, 'min_samples_leaf': 46, 'l2_regularization': 0.00017271920937052574, 'max_bins': 185}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:27:59,975] Trial 23 finished with value: 0.44810839432575794 and parameters: {'learning_rate': 0.09633643842482383, 'max_iter': 588, 'max_leaf_nodes': 900, 'max_depth': 8, 'min_samples_leaf': 79, 'l2_regularization': 0.0020992794939752313, 'max_bins': 205}. Best is trial 16 with value: 0.44966332375186335.
[I 2025-09-21 01:28:02,328] Trial 24 finished with value: 0.4518218182495888 and parameters: {'learning_rate': 0.04247056107813091, 'max_iter': 538, 'max_leaf_nodes': 320, 'max_depth': 7, 'min_samples_leaf': 100, 'l2_regularization': 0.005441154002659272, 'max_bins': 207}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:05,002] Trial 25 finished with value: 0.4457252915798932 and parameters: {'learning_rate': 0.04163986858874114, 'max_iter': 423, 'max_leaf_nodes': 300, 'max_depth': 6, 'min_samples_leaf': 144, 'l2_regularization': 6.710242586881765e-06, 'max_bins': 222}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:06,764] Trial 26 finished with value: 0.4429759141594265 and parameters: {'learning_rate': 0.07519020818200013, 'max_iter': 1089, 'max_leaf_nodes': 360, 'max_depth': 7, 'min_samples_leaf': 163, 'l2_regularization': 0.056799192505577065, 'max_bins': 236}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:08,908] Trial 27 finished with value: 0.4473918586762801 and parameters: {'learning_rate': 0.04346190178940336, 'max_iter': 2759, 'max_leaf_nodes': 720, 'max_depth': 10, 'min_samples_leaf': 96, 'l2_regularization': 0.006261025045589711, 'max_bins': 173}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:10,706] Trial 28 finished with value: 0.43815458562787424 and parameters: {'learning_rate': 0.031485552775758, 'max_iter': 134, 'max_leaf_nodes': 1240, 'max_depth': 5, 'min_samples_leaf': 134, 'l2_regularization': 0.00020067926784126034, 'max_bins': 200}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:12,074] Trial 29 finished with value: 0.4447296396044894 and parameters: {'learning_rate': 0.15476205701443924, 'max_iter': 2357, 'max_leaf_nodes': 260, 'max_depth': 7, 'min_samples_leaf': 25, 'l2_regularization': 0.6670131777444449, 'max_bins': 156}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:12,825] Trial 30 finished with value: 0.4306405176677598 and parameters: {'learning_rate': 0.2917427363841763, 'max_iter': 950, 'max_leaf_nodes': 500, 'max_depth': 11, 'min_samples_leaf': 160, 'l2_regularization': 8.822869199382964e-06, 'max_bins': 212}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:14,108] Trial 31 finished with value: 0.4432131242234094 and parameters: {'learning_rate': 0.09609447687923851, 'max_iter': 681, 'max_leaf_nodes': 860, 'max_depth': 8, 'min_samples_leaf': 102, 'l2_regularization': 0.004903105183029196, 'max_bins': 195}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:15,835] Trial 32 finished with value: 0.44950112411680854 and parameters: {'learning_rate': 0.06610170749508154, 'max_iter': 444, 'max_leaf_nodes': 900, 'max_depth': 8, 'min_samples_leaf': 84, 'l2_regularization': 0.0022605663929301323, 'max_bins': 176}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:17,661] Trial 33 finished with value: 0.448064755821874 and parameters: {'learning_rate': 0.05814255295555452, 'max_iter': 323, 'max_leaf_nodes': 580, 'max_depth': 9, 'min_samples_leaf': 88, 'l2_regularization': 0.0002991958670999678, 'max_bins': 132}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:19,297] Trial 34 finished with value: 0.44618307401524715 and parameters: {'learning_rate': 0.07541366516171705, 'max_iter': 458, 'max_leaf_nodes': 760, 'max_depth': 7, 'min_samples_leaf': 119, 'l2_regularization': 0.037964772860774064, 'max_bins': 174}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:22,576] Trial 35 finished with value: 0.44255861477216357 and parameters: {'learning_rate': 0.03664150564133877, 'max_iter': 861, 'max_leaf_nodes': 200, 'max_depth': 5, 'min_samples_leaf': 41, 'l2_regularization': 0.0026693842239720747, 'max_bins': 167}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:24,278] Trial 36 finished with value: 0.44773776451228625 and parameters: {'learning_rate': 0.06771789913026575, 'max_iter': 1260, 'max_leaf_nodes': 1400, 'max_depth': 8, 'min_samples_leaf': 69, 'l2_regularization': 7.810119143309551, 'max_bins': 194}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:25,445] Trial 37 finished with value: 0.44466928629582103 and parameters: {'learning_rate': 0.10976252864364888, 'max_iter': 645, 'max_leaf_nodes': 1820, 'max_depth': 9, 'min_samples_leaf': 105, 'l2_regularization': 0.1578118079905398, 'max_bins': 216}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:27,736] Trial 38 finished with value: 0.4328862860267456 and parameters: {'learning_rate': 0.08426560602210315, 'max_iter': 1865, 'max_leaf_nodes': 1180, 'max_depth': 3, 'min_samples_leaf': 148, 'l2_regularization': 0.02035785136662644, 'max_bins': 180}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:29,789] Trial 39 finished with value: 0.44762829957411404 and parameters: {'learning_rate': 0.04815681025533597, 'max_iter': 2810, 'max_leaf_nodes': 20, 'max_depth': 11, 'min_samples_leaf': 65, 'l2_regularization': 1.8525243932163005e-05, 'max_bins': 151}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:30,951] Trial 40 finished with value: 0.4409508448991527 and parameters: {'learning_rate': 0.14647738589981127, 'max_iter': 335, 'max_leaf_nodes': 600, 'max_depth': 10, 'min_samples_leaf': 131, 'l2_regularization': 2.0681014886205296e-06, 'max_bins': 134}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:32,251] Trial 41 finished with value: 0.44673930293605146 and parameters: {'learning_rate': 0.10939046215675335, 'max_iter': 613, 'max_leaf_nodes': 840, 'max_depth': 8, 'min_samples_leaf': 81, 'l2_regularization': 0.0006781965671783909, 'max_bins': 203}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:36,159] Trial 42 finished with value: 0.4482958805150772 and parameters: {'learning_rate': 0.028281551314508074, 'max_iter': 520, 'max_leaf_nodes': 940, 'max_depth': 6, 'min_samples_leaf': 83, 'l2_regularization': 0.002464588956588091, 'max_bins': 210}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:37,829] Trial 43 finished with value: 0.45142417411459324 and parameters: {'learning_rate': 0.027141098524758052, 'max_iter': 103, 'max_leaf_nodes': 1520, 'max_depth': 7, 'min_samples_leaf': 89, 'l2_regularization': 0.015402842909136695, 'max_bins': 243}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:40,577] Trial 44 finished with value: 0.4470614716725012 and parameters: {'learning_rate': 0.012057053965141843, 'max_iter': 163, 'max_leaf_nodes': 1680, 'max_depth': 7, 'min_samples_leaf': 113, 'l2_regularization': 7.807416980163017e-05, 'max_bins': 245}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:42,546] Trial 45 finished with value: 0.44501323216978966 and parameters: {'learning_rate': 0.056453845768946534, 'max_iter': 299, 'max_leaf_nodes': 1960, 'max_depth': 6, 'min_samples_leaf': 90, 'l2_regularization': 0.020407764239841614, 'max_bins': 192}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:43,514] Trial 46 finished with value: 0.4440610511951345 and parameters: {'learning_rate': 0.17087798602030746, 'max_iter': 797, 'max_leaf_nodes': 460, 'max_depth': 7, 'min_samples_leaf': 58, 'l2_regularization': 0.010225130205108013, 'max_bins': 178}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:46,873] Trial 47 finished with value: 0.44725436662506424 and parameters: {'learning_rate': 0.027430770778718187, 'max_iter': 2174, 'max_leaf_nodes': 1520, 'max_depth': 9, 'min_samples_leaf': 126, 'l2_regularization': 0.00014203873783824345, 'max_bins': 239}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:48,559] Trial 48 finished with value: 0.44559123310598003 and parameters: {'learning_rate': 0.06632412991179419, 'max_iter': 2389, 'max_leaf_nodes': 2080, 'max_depth': 6, 'min_samples_leaf': 100, 'l2_regularization': 0.0006232778523992839, 'max_bins': 247}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:51,168] Trial 49 finished with value: 0.42677614564149485 and parameters: {'learning_rate': 0.010238213371461524, 'max_iter': 236, 'max_leaf_nodes': 1300, 'max_depth': 4, 'min_samples_leaf': 113, 'l2_regularization': 0.06702536983046976, 'max_bins': 105}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:53,417] Trial 50 finished with value: 0.44219191958539145 and parameters: {'learning_rate': 0.04985332860822653, 'max_iter': 1503, 'max_leaf_nodes': 2940, 'max_depth': 12, 'min_samples_leaf': 154, 'l2_regularization': 0.510365616749533, 'max_bins': 164}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:57,714] Trial 51 finished with value: 0.44820052213503353 and parameters: {'learning_rate': 0.02451018288146342, 'max_iter': 485, 'max_leaf_nodes': 1080, 'max_depth': 6, 'min_samples_leaf': 88, 'l2_regularization': 0.0038170291070721, 'max_bins': 212}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:28:59,165] Trial 52 finished with value: 0.4424184134976138 and parameters: {'learning_rate': 0.030981901367626373, 'max_iter': 113, 'max_leaf_nodes': 1460, 'max_depth': 5, 'min_samples_leaf': 72, 'l2_regularization': 0.0010321865565063738, 'max_bins': 220}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:03,023] Trial 53 finished with value: 0.448159152011734 and parameters: {'learning_rate': 0.0260356353324563, 'max_iter': 2966, 'max_leaf_nodes': 2680, 'max_depth': 7, 'min_samples_leaf': 173, 'l2_regularization': 0.01356084859448723, 'max_bins': 231}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:05,576] Trial 54 finished with value: 0.4502352951623608 and parameters: {'learning_rate': 0.03662160924774511, 'max_iter': 453, 'max_leaf_nodes': 1680, 'max_depth': 8, 'min_samples_leaf': 109, 'l2_regularization': 0.0017746916321142, 'max_bins': 190}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:08,143] Trial 55 finished with value: 0.4478536404910085 and parameters: {'learning_rate': 0.04010428508883373, 'max_iter': 712, 'max_leaf_nodes': 1620, 'max_depth': 8, 'min_samples_leaf': 106, 'l2_regularization': 0.00045248358716110294, 'max_bins': 190}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:09,599] Trial 56 finished with value: 0.44566111645213846 and parameters: {'learning_rate': 0.08294554973223292, 'max_iter': 414, 'max_leaf_nodes': 1720, 'max_depth': 9, 'min_samples_leaf': 139, 'l2_regularization': 7.500335942576046e-05, 'max_bins': 199}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:11,702] Trial 57 finished with value: 0.45065916386954796 and parameters: {'learning_rate': 0.05442072337299227, 'max_iter': 261, 'max_leaf_nodes': 1840, 'max_depth': 8, 'min_samples_leaf': 96, 'l2_regularization': 2.5003347302226983e-05, 'max_bins': 183}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:13,289] Trial 58 finished with value: 0.4477971915011392 and parameters: {'learning_rate': 0.08371102645845174, 'max_iter': 341, 'max_leaf_nodes': 1880, 'max_depth': 9, 'min_samples_leaf': 96, 'l2_regularization': 8.928881277281747e-08, 'max_bins': 183}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:15,165] Trial 59 finished with value: 0.44823095616428565 and parameters: {'learning_rate': 0.053342332805855694, 'max_iter': 1326, 'max_leaf_nodes': 2220, 'max_depth': 8, 'min_samples_leaf': 118, 'l2_regularization': 3.0974420602596736e-05, 'max_bins': 170}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:16,237] Trial 60 finished with value: 0.4414460692780159 and parameters: {'learning_rate': 0.1420751236333855, 'max_iter': 262, 'max_leaf_nodes': 2020, 'max_depth': 10, 'min_samples_leaf': 107, 'l2_regularization': 0.007911523936385655, 'max_bins': 188}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:20,875] Trial 61 finished with value: 0.4473855936654564 and parameters: {'learning_rate': 0.017861936819313855, 'max_iter': 533, 'max_leaf_nodes': 100, 'max_depth': 7, 'min_samples_leaf': 128, 'l2_regularization': 0.0012963089726459138, 'max_bins': 160}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:22,501] Trial 62 finished with value: 0.44899620996208717 and parameters: {'learning_rate': 0.06590455473983405, 'max_iter': 100, 'max_leaf_nodes': 1800, 'max_depth': 8, 'min_samples_leaf': 98, 'l2_regularization': 1.83685481874032e-05, 'max_bins': 176}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:23,541] Trial 63 finished with value: 0.441909799097471 and parameters: {'learning_rate': 0.2182081725257557, 'max_iter': 411, 'max_leaf_nodes': 1580, 'max_depth': 7, 'min_samples_leaf': 78, 'l2_regularization': 0.00023865724057590683, 'max_bins': 182}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:26,091] Trial 64 finished with value: 0.4494117335556919 and parameters: {'learning_rate': 0.0405739341679877, 'max_iter': 2692, 'max_leaf_nodes': 1380, 'max_depth': 8, 'min_samples_leaf': 90, 'l2_regularization': 0.0001395469427134599, 'max_bins': 207}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:30,291] Trial 65 finished with value: 0.4496674518485559 and parameters: {'learning_rate': 0.019589613561266015, 'max_iter': 1029, 'max_leaf_nodes': 1100, 'max_depth': 10, 'min_samples_leaf': 46, 'l2_regularization': 3.0021231364214333e-06, 'max_bins': 197}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:33,032] Trial 66 finished with value: 0.4507710690792436 and parameters: {'learning_rate': 0.036513982819219415, 'max_iter': 961, 'max_leaf_nodes': 1160, 'max_depth': 10, 'min_samples_leaf': 27, 'l2_regularization': 2.936949069126323e-06, 'max_bins': 199}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:37,808] Trial 67 finished with value: 0.4511559818613895 and parameters: {'learning_rate': 0.02014507537225522, 'max_iter': 978, 'max_leaf_nodes': 1140, 'max_depth': 10, 'min_samples_leaf': 36, 'l2_regularization': 4.252114959665531e-07, 'max_bins': 198}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:42,137] Trial 68 finished with value: 0.4500628433904275 and parameters: {'learning_rate': 0.01978684814876274, 'max_iter': 1107, 'max_leaf_nodes': 1140, 'max_depth': 11, 'min_samples_leaf': 37, 'l2_regularization': 4.1095655431669626e-07, 'max_bins': 199}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:45,156] Trial 69 finished with value: 0.45016710014899247 and parameters: {'learning_rate': 0.034459553146963195, 'max_iter': 1124, 'max_leaf_nodes': 1240, 'max_depth': 11, 'min_samples_leaf': 37, 'l2_regularization': 4.196106815747503e-07, 'max_bins': 202}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:48,218] Trial 70 finished with value: 0.4460738852883194 and parameters: {'learning_rate': 0.0353940566259765, 'max_iter': 897, 'max_leaf_nodes': 1280, 'max_depth': 12, 'min_samples_leaf': 20, 'l2_regularization': 2.2082096407593237e-07, 'max_bins': 217}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:50,726] Trial 71 finished with value: 0.448635494656477 and parameters: {'learning_rate': 0.04580876325496763, 'max_iter': 1404, 'max_leaf_nodes': 1480, 'max_depth': 11, 'min_samples_leaf': 31, 'l2_regularization': 5.636822463977303e-07, 'max_bins': 204}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:55,501] Trial 72 finished with value: 0.4480742310608122 and parameters: {'learning_rate': 0.017716458412620872, 'max_iter': 1210, 'max_leaf_nodes': 1200, 'max_depth': 11, 'min_samples_leaf': 26, 'l2_regularization': 3.429403411948654e-07, 'max_bins': 199}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:29:58,581] Trial 73 finished with value: 0.4487376557664559 and parameters: {'learning_rate': 0.03526382051252655, 'max_iter': 998, 'max_leaf_nodes': 1640, 'max_depth': 12, 'min_samples_leaf': 36, 'l2_regularization': 1.4228296144513823e-07, 'max_bins': 225}. Best is trial 24 with value: 0.4518218182495888.
[I 2025-09-21 01:30:00,643] Trial 74 finished with value: 0.45227187120238294 and parameters: {'learning_rate': 0.053357313464004186, 'max_iter': 1200, 'max_leaf_nodes': 1120, 'max_depth': 10, 'min_samples_leaf': 40, 'l2_regularization': 4.5193224454909995e-08, 'max_bins': 193}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:02,472] Trial 75 finished with value: 0.45117944408660815 and parameters: {'learning_rate': 0.0589876069776919, 'max_iter': 1193, 'max_leaf_nodes': 1400, 'max_depth': 10, 'min_samples_leaf': 48, 'l2_regularization': 4.044053495910081e-08, 'max_bins': 208}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:04,532] Trial 76 finished with value: 0.45000654140773433 and parameters: {'learning_rate': 0.057373686700601466, 'max_iter': 1613, 'max_leaf_nodes': 1760, 'max_depth': 10, 'min_samples_leaf': 42, 'l2_regularization': 1.3098181574177022e-08, 'max_bins': 194}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:06,150] Trial 77 finished with value: 0.4481276225977773 and parameters: {'learning_rate': 0.07287111018718173, 'max_iter': 752, 'max_leaf_nodes': 1380, 'max_depth': 10, 'min_samples_leaf': 52, 'l2_regularization': 4.543156418937283e-08, 'max_bins': 188}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:08,505] Trial 78 finished with value: 0.45133267638577645 and parameters: {'learning_rate': 0.04728205209641608, 'max_iter': 825, 'max_leaf_nodes': 2140, 'max_depth': 9, 'min_samples_leaf': 48, 'l2_regularization': 3.6478063499641364e-08, 'max_bins': 214}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:10,408] Trial 79 finished with value: 0.4507594473954839 and parameters: {'learning_rate': 0.050273154841273524, 'max_iter': 821, 'max_leaf_nodes': 2440, 'max_depth': 9, 'min_samples_leaf': 51, 'l2_regularization': 6.03593194108357e-08, 'max_bins': 217}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:12,742] Trial 80 finished with value: 0.44515412282147215 and parameters: {'learning_rate': 0.04736040166701908, 'max_iter': 820, 'max_leaf_nodes': 2600, 'max_depth': 13, 'min_samples_leaf': 51, 'l2_regularization': 2.1412841371409727e-08, 'max_bins': 217}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:14,498] Trial 81 finished with value: 0.4508683656950006 and parameters: {'learning_rate': 0.05845948627410413, 'max_iter': 1170, 'max_leaf_nodes': 2360, 'max_depth': 9, 'min_samples_leaf': 42, 'l2_regularization': 6.035928711247453e-08, 'max_bins': 208}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:16,085] Trial 82 finished with value: 0.4482149454439203 and parameters: {'learning_rate': 0.0657200601261816, 'max_iter': 1190, 'max_leaf_nodes': 2480, 'max_depth': 10, 'min_samples_leaf': 59, 'l2_regularization': 3.167447510306652e-08, 'max_bins': 207}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:17,580] Trial 83 finished with value: 0.44946955958443724 and parameters: {'learning_rate': 0.07855074283430447, 'max_iter': 907, 'max_leaf_nodes': 2420, 'max_depth': 9, 'min_samples_leaf': 47, 'l2_regularization': 7.6953731197265e-08, 'max_bins': 208}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:19,539] Trial 84 finished with value: 0.45014061044730996 and parameters: {'learning_rate': 0.06157334697553249, 'max_iter': 968, 'max_leaf_nodes': 2280, 'max_depth': 10, 'min_samples_leaf': 27, 'l2_regularization': 1.1529684292256703e-08, 'max_bins': 231}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:21,670] Trial 85 finished with value: 0.4494937118966333 and parameters: {'learning_rate': 0.042928536991277196, 'max_iter': 1275, 'max_leaf_nodes': 2140, 'max_depth': 9, 'min_samples_leaf': 64, 'l2_regularization': 5.34976268217675e-08, 'max_bins': 225}. Best is trial 74 with value: 0.45227187120238294.
[I 2025-09-21 01:30:23,992] Trial 86 finished with value: 0.4525374406583695 and parameters: {'learning_rate': 0.050623168535997125, 'max_iter': 1390, 'max_leaf_nodes': 2260, 'max_depth': 9, 'min_samples_leaf': 41, 'l2_regularization': 1.1298645707113427e-07, 'max_bins': 213}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:27,499] Trial 87 finished with value: 0.4510532364115852 and parameters: {'learning_rate': 0.027832370091484662, 'max_iter': 1383, 'max_leaf_nodes': 2360, 'max_depth': 10, 'min_samples_leaf': 43, 'l2_regularization': 1.4766580940953745e-07, 'max_bins': 221}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:28,905] Trial 88 finished with value: 0.4494023394521907 and parameters: {'learning_rate': 0.09001805724165837, 'max_iter': 1422, 'max_leaf_nodes': 2320, 'max_depth': 9, 'min_samples_leaf': 42, 'l2_regularization': 1.1079648401290462e-07, 'max_bins': 212}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:32,368] Trial 89 finished with value: 0.4466989024289387 and parameters: {'learning_rate': 0.025775035275946028, 'max_iter': 1587, 'max_leaf_nodes': 2820, 'max_depth': 10, 'min_samples_leaf': 57, 'l2_regularization': 1.758432400553505e-07, 'max_bins': 238}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:39,977] Trial 90 finished with value: 0.445140410604401 and parameters: {'learning_rate': 0.010411933280841985, 'max_iter': 1163, 'max_leaf_nodes': 2360, 'max_depth': 11, 'min_samples_leaf': 32, 'l2_regularization': 2.903428126775598e-08, 'max_bins': 221}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:43,270] Trial 91 finished with value: 0.44974710751588154 and parameters: {'learning_rate': 0.03083273280555556, 'max_iter': 1275, 'max_leaf_nodes': 2120, 'max_depth': 10, 'min_samples_leaf': 46, 'l2_regularization': 1.3449488426508309e-06, 'max_bins': 213}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:45,046] Trial 92 finished with value: 0.447297632555461 and parameters: {'learning_rate': 0.06053451854056176, 'max_iter': 1337, 'max_leaf_nodes': 2580, 'max_depth': 10, 'min_samples_leaf': 23, 'l2_regularization': 2.3370447511220006e-07, 'max_bins': 255}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:47,390] Trial 93 finished with value: 0.452216702796579 and parameters: {'learning_rate': 0.0433075677016712, 'max_iter': 1060, 'max_leaf_nodes': 2200, 'max_depth': 9, 'min_samples_leaf': 39, 'l2_regularization': 7.379442494835273e-07, 'max_bins': 202}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:49,067] Trial 94 finished with value: 0.4491483790607006 and parameters: {'learning_rate': 0.07247736285392223, 'max_iter': 1057, 'max_leaf_nodes': 2200, 'max_depth': 9, 'min_samples_leaf': 40, 'l2_regularization': 9.34526207566645e-07, 'max_bins': 228}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:51,345] Trial 95 finished with value: 0.45221560984852793 and parameters: {'learning_rate': 0.0456599633851376, 'max_iter': 1485, 'max_leaf_nodes': 1960, 'max_depth': 9, 'min_samples_leaf': 34, 'l2_regularization': 3.5038090694483294e-08, 'max_bins': 209}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:53,866] Trial 96 finished with value: 0.4498540833211532 and parameters: {'learning_rate': 0.04377538950895976, 'max_iter': 1739, 'max_leaf_nodes': 1980, 'max_depth': 9, 'min_samples_leaf': 34, 'l2_regularization': 3.976338772161556e-08, 'max_bins': 250}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:56,138] Trial 97 finished with value: 0.45109117385858893 and parameters: {'learning_rate': 0.04942678277861792, 'max_iter': 1442, 'max_leaf_nodes': 2060, 'max_depth': 10, 'min_samples_leaf': 48, 'l2_regularization': 2.2614253703169104e-08, 'max_bins': 221}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:58,331] Trial 98 finished with value: 0.4462808462721469 and parameters: {'learning_rate': 0.05094713353561294, 'max_iter': 1219, 'max_leaf_nodes': 2080, 'max_depth': 11, 'min_samples_leaf': 54, 'l2_regularization': 1.7867615901382662e-08, 'max_bins': 234}. Best is trial 86 with value: 0.4525374406583695.
[I 2025-09-21 01:30:59,665] Trial 99 finished with value: 0.44661243363114866 and parameters: {'learning_rate': 0.1022366733104485, 'max_iter': 1485, 'max_leaf_nodes': 1940, 'max_depth': 9, 'min_samples_leaf': 69, 'l2_regularization': 2.106805688003544e-08, 'max_bins': 242}. Best is trial 86 with value: 0.4525374406583695.

--- Hyperparameter Optimization Complete ---

Best trial:
  Value: 0.4525374406583695
  Params: 
    learning_rate: 0.050623168535997125
    max_iter: 1390
    max_leaf_nodes: 2260
    max_depth: 9
    min_samples_leaf: 41
    l2_regularization: 1.1298645707113427e-07
    max_bins: 213

Training Best Models

The fine-tuned XGBoost and LightGBM models achieved the best F1-Score at around 0.478. We will now build the entire pipeline to evaluate performance and compare the two of them. We will also check feature importance plots to help us decide which model has the most robust and consistent performance.

# Retrieving the best set of hyperparameters
best_lgbm_params = study_lgbm.best_params

final_lgbm_model = LGBMClassifier(**best_lgbm_params, random_state=seed, n_jobs=-1, verbose=-1)

# Creating the final pipeline with the best LightGBM model
final_lgbm_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor_engineered),
    ('classifier', final_lgbm_model)
])

# Training the final model
print("\n--- Training Final LightGBM Model with Optimized Hyperparameters ---\n")
start_time = time.time()
final_lgbm_pipeline.fit(X_train, y_train)
end_time = time.time()
training_time = end_time - start_time
print(f"Final LightGBM model trained in {training_time:.2f} seconds.")
y_pred_final = final_lgbm_pipeline.predict(X_test)
y_proba_final = final_lgbm_pipeline.predict_proba(X_test)[:, 1]

# Evaluating the final model
print("\n--- Final Model Evaluation ---\n")
print("Classification Report:\n")
print(classification_report(y_test, y_pred_final, target_names=['no', 'yes']))
plot_confusion_matrix(y_test, y_pred_final, "Final Tuned LightGBM Model")

final_lgbm_metrics = {
    'Precision': precision_score(y_test, y_pred_final),
    'Recall': recall_score(y_test, y_pred_final),
    'F1-Score': f1_score(y_test, y_pred_final),
    'ROC AUC': roc_auc_score(y_test, y_proba_final),
    'Training Time (s)': training_time
}
final_lgbm_results_df = pd.DataFrame([final_lgbm_metrics], index=['Final Tuned LightGBM Model'])
print("\nFinal Model Performance:\n")
display(final_lgbm_results_df.style.format({'Precision': '{:.2f}', 'Recall': '{:.2f}', 'F1-Score': '{:.2f}',
                                       'ROC AUC': '{:.2f}', 'Training Time (s)': '{:.2f}'}))
# Plotting ROC Curve for the final model
fpr, tpr, _ = roc_curve(y_test, y_proba_final, pos_label=1)
roc_auc = auc(fpr, tpr)

fig = go.Figure()

fig.add_trace(go.Scatter(
    x=fpr, 
    y=tpr,
    mode='lines',
    line=dict(color='#E85B5B', width=2),
    name=f"Final Tuned Model (AUC = {roc_auc:.2f})"
))
fig.add_trace(go.Scatter(
    x=[0, 1], 
    y=[0, 1],
    mode='lines',
    line=dict(color='black', width=2, dash='dash'),
    name='Random Guess'
))

fig.update_layout(
    title='<b>ROC Curve for Final Tuned LightGBM Model</b>',
    xaxis_title='False Positive Rate',
    yaxis_title='True Positive Rate',
    xaxis=dict(range=[0.0, 1.0]),
    yaxis=dict(range=[0.0, 1.05]),
    width=700,
    height=700,
    plot_bgcolor='#f5f7f6',
    paper_bgcolor='#f5f7f6',
    legend=dict(x=0.55, y=0.1),
    xaxis_scaleanchor="y",
    yaxis_scaleratio=1
)

fig.show()
    

--- Training Final LightGBM Model with Optimized Hyperparameters ---

Final LightGBM model trained in 7.14 seconds.

--- Final Model Evaluation ---

Classification Report:

              precision    recall  f1-score   support

          no       0.94      0.91      0.92      7981
         yes       0.44      0.54      0.49      1058

    accuracy                           0.87      9039
   macro avg       0.69      0.73      0.71      9039
weighted avg       0.88      0.87      0.87      9039

Final Model Performance:

	Precision	Recall	F1-Score	ROC AUC	Training Time (s)
Final Tuned LightGBM Model	0.44	0.54	0.49	0.81	7.14

# Retrieving the best set of hyperparameters
best_xgb_params = study_xgb.best_params

final_xgb_model = XGBClassifier(**best_xgb_params, random_state=seed, n_jobs=-1, verbose=-1)

# Creating the final pipeline with the best XGBoost model
final_xgb_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor_engineered),
    ('classifier', final_xgb_model)
])

# Training the final model
print("\n--- Training Final XGBoost Model with Optimized Hyperparameters ---\n")
start_time = time.time()
final_xgb_pipeline.fit(X_train, y_train)
end_time = time.time()
training_time = end_time - start_time
print(f"Final XGBoost model trained in {training_time:.2f} seconds.")
y_pred_final = final_xgb_pipeline.predict(X_test)
y_proba_final = final_xgb_pipeline.predict_proba(X_test)[:, 1]

# Evaluating the final model
print("\n--- Final Model Evaluation ---\n")
print("Classification Report:\n")
print(classification_report(y_test, y_pred_final, target_names=['no', 'yes']))
plot_confusion_matrix(y_test, y_pred_final, "Final Tuned XGBoost Model")

final_xgb_metrics = {
    'Precision': precision_score(y_test, y_pred_final),
    'Recall': recall_score(y_test, y_pred_final),
    'F1-Score': f1_score(y_test, y_pred_final),
    'ROC AUC': roc_auc_score(y_test, y_proba_final),
    'Training Time (s)': training_time
}
final_xgb_results_df = pd.DataFrame([final_xgb_metrics], index=['Final Tuned XGBoost Model'])
print("\nFinal Model Performance:\n")
display(final_xgb_results_df.style.format({'Precision': '{:.2f}', 'Recall': '{:.2f}', 'F1-Score': '{:.2f}',
                                       'ROC AUC': '{:.2f}', 'Training Time (s)': '{:.2f}'}))
# Plotting ROC Curve for the final model
fpr, tpr, _ = roc_curve(y_test, y_proba_final, pos_label=1)
roc_auc = auc(fpr, tpr)

fig = go.Figure()

fig.add_trace(go.Scatter(
    x=fpr, 
    y=tpr,
    mode='lines',
    line=dict(color='#E85B5B', width=2),
    name=f"Final Tuned Model (AUC = {roc_auc:.2f})"
))
fig.add_trace(go.Scatter(
    x=[0, 1], 
    y=[0, 1],
    mode='lines',
    line=dict(color='black', width=2, dash='dash'),
    name='Random Guess'
))

fig.update_layout(
    title='<b>ROC Curve for Final Tuned XGBoost Model</b>',
    xaxis_title='False Positive Rate',
    yaxis_title='True Positive Rate',
    xaxis=dict(range=[0.0, 1.0]),
    yaxis=dict(range=[0.0, 1.05]),
    width=700,
    height=700,
    plot_bgcolor='#f5f7f6',
    paper_bgcolor='#f5f7f6',
    legend=dict(x=0.55, y=0.1),
    xaxis_scaleanchor="y",
    yaxis_scaleratio=1
)

fig.show()
    

--- Training Final XGBoost Model with Optimized Hyperparameters ---

Final XGBoost model trained in 2.63 seconds.

--- Final Model Evaluation ---

Classification Report:

              precision    recall  f1-score   support

          no       0.94      0.91      0.92      7981
         yes       0.44      0.57      0.50      1058

    accuracy                           0.87      9039
   macro avg       0.69      0.74      0.71      9039
weighted avg       0.88      0.87      0.87      9039

Final Model Performance:

	Precision	Recall	F1-Score	ROC AUC	Training Time (s)
Final Tuned XGBoost Model	0.44	0.57	0.50	0.81	2.63

Both the fine-tuned versions of the XGBoost and LightGBM achieve close results in F1-Score in the final training(0.50 and 0.49 respectively). However, the LightGBM took a bit longer to train compared to the XGBoost model (7.14 seconds vs 2.63). Although this difference in training time could get us inclined to deploy the XGBoost model, we should investigate further to understand how both models are using the features to define an outcome for the target variable.

Feature Importance

Before deciding which model to pick, it is a good idea to visualize feature importances to understand which data points are most influentional in their decision-making. This also helps us assess how reliable and robust the model is in its predictions.

# Computing Feature Importance
importances = final_xgb_pipeline.named_steps['classifier'].feature_importances_
feature_names = final_xgb_pipeline.named_steps['preprocessor'].get_feature_names_out()
feature_importances = pd.Series(importances, index=feature_names).sort_values(ascending=False).head(20)

# Plotting Feature Importances
fig = go.Figure(go.Bar(
    x=feature_importances.values,
    y=feature_importances.index,
    orientation='h',
    marker=dict(
        color=feature_importances.values,
        colorscale='Viridis'
    )
))

fig.update_layout(
    title={
        'text': '<b>Feature Importances - Final Tuned XGBoost Model</b>',
        'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
    },
    xaxis_title='Importance',
    yaxis_title='Features',
    yaxis=dict(autorange='reversed'),
    plot_bgcolor='#f5f7f6',
    paper_bgcolor='#f5f7f6',
    height=700,
    width=1000
)

fig.show()

• Overliance in One Single Feature
  • The cat__poutcome_success feature has an importance score that is more than double the next most important feature (cat__contact_unknown) and orders of magnitude larger than most others. This is a red flag, because it means the model has become overreliant on one single type of input.
• Potential Risks
  • The model likely learned a very simple rule: If poutcome is success, predict yes. This is a sign that the model have overfit to the specific group of customer with a known successful past campaign.

# Computing Feature Importance
importances = final_lgbm_pipeline.named_steps['classifier'].feature_importances_
feature_names = final_lgbm_pipeline.named_steps['preprocessor'].get_feature_names_out()
feature_importances = pd.Series(importances, index=feature_names).sort_values(ascending=False).head(20)

# Plotting Feature Importances
fig = go.Figure(go.Bar(
    x=feature_importances.values,
    y=feature_importances.index,
    orientation='h',
    marker=dict(
        color=feature_importances.values,
        colorscale='Viridis'
    )
))

fig.update_layout(
    title={
        'text': '<b>Feature Importances - Final Tuned LGBM Model</b>',
        'y':0.95, 'x':0.5, 'xanchor': 'center', 'yanchor': 'top'
    },
    xaxis_title='Importance',
    yaxis_title='Features',
    yaxis=dict(autorange='reversed'),
    plot_bgcolor='#f5f7f6',
    paper_bgcolor='#f5f7f6',
    height=700,
    width=1000
)

fig.show()

The feature importance plot for the LightGBM model revelas some interesting insights:

• Client Financial Profile: The client's financial situation is one of the most dominant factor. balance is the single most important feature. The model also uses information about existing housing and personal loans, supporting our hypothesis that clients with fewer financial liabilities are a key target group.
  
  
• Campaign Strategy & Timing: Our feature engineering efforts have proven to be effective. The new cyclical day features (day_sin and day_cos) have emerged as the 2nd and 4th most important predictors overall, confirming that there is a powerful pattern related to the time of the month. The campaign contact count, past engagement (pdays), and cyclical month features also remain key strategic and behavioral indicators.
  
  
• Client Demographics: Client demographics are a top-tier predictor, with age* ranking as the third most powerful feature in the model. Other details such as maritalstatus,educationlevel, andjob` type continue to contribute significantly, allowing the model to build a detailed and nuanced profile of a likely subscriber.

Saving Model

After evaluating the feature importance plots, we can conclude that the fine-tuned LightGBM model is more reliable and more robust than the XGBoost. Not only the model places importance across several features in a more balance way, but it has also learned to use the new cyclical time features as a powerful signal, which should help it generalize better in production.

The final step of this notebook is to save our final model so it can be used in other applications. We'll use the joblib library to serialize our entire final_lgbm_pipeline object to a file. Saving the complete pipeline is crucial, as it ensures the exact same preprocessing steps are applied to new data during prediction, guaranteeing consistent results. This file can then be loaded to make live predictions without needing to retrain the model.

# Displaying final pipeline
final_lgbm_pipeline

Pipeline(steps=[('preprocessor',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('cat',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  Index(['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
       'poutcome'],
      dtype='object')),
                                                 ('num', PowerTransformer(),
                                                  ['age', 'balance', 'campaign',
                                                   'pdays', 'previous']),
                                                 ('cyclical', 'passthrough',
                                                  ['month_sin', '...
                ('classifier',
                 LGBMClassifier(boosting_type='dart',
                                colsample_bytree=0.6045884605959665,
                                learning_rate=0.011933663386885873,
                                max_depth=12, min_child_samples=99,
                                n_estimators=200, n_jobs=-1, num_leaves=2980,
                                random_state=42,
                                reg_alpha=1.0023526200624856e-08,
                                reg_lambda=1.246535480598537e-08,
                                scale_pos_weight=7.009971471773609,
                                subsample=0.976117178865086, verbose=-1))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

# Saving model
model_dir = '../models'
model_name = 'lgbm'
file_name = f'final_{model_name}_pipeline_v2.joblib'
file_path = os.path.join(model_dir, file_name)

os.makedirs(model_dir, exist_ok=True)
joblib.dump(final_lgbm_pipeline, file_path)

print(f"\nModel saved to {file_path}")

Model saved to ../models/final_lgbm_pipeline_v2.joblib

---

---

Conclusion

We have successfully developed a machine learning model to improve the efficiency of a bank's telemarketing campaigns. We followed a Data Science workflow, beginning with an Exploratory Data Analysis, then moving on to Data Pipeline & Preprocessing, and finally Model Selection and Fine-Tuning.

Our final model, a fine-tuned LightGBM Classifier, achieved a F1-Score of 0.49 and a ROC AUC of 0.81 on the hold-out test set. This represents an improvement over our baseline and provides the bank with a tool to identify and prioritize high-potential clients.

Our analysis confirmed the critical importance of client demographic (age, job), financial profile (balance, existing loans), and campaign timing (day and month of conduction). With these patterns, the final model can help the bank focus its resources more effectively, reducing costs associated with contacting uninterested clients, and ultimately increase its term deposit subscription rate. The saved pipeline is now ready for deployment in a production environment or an interactive application.

---

---